Oct 30 17:55:28.062909 ip-10-0-133-124 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 17:55:28.062920 ip-10-0-133-124 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 17:55:28.062928 ip-10-0-133-124 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 17:55:28.063143 ip-10-0-133-124 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 17:55:38.312191 ip-10-0-133-124 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 17:55:38.312204 ip-10-0-133-124 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7c3d372cac0e4d9e8931432f273512d9 -- Oct 30 17:57:24.601586 ip-10-0-133-124 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:57:24.994531 ip-10-0-133-124 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:24.994531 ip-10-0-133-124 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:57:24.994531 ip-10-0-133-124 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:24.994531 ip-10-0-133-124 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:24.999280 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999185 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:24.999280 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999279 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:24.999280 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999283 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999288 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999291 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999295 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999298 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999301 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999304 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999306 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999309 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999312 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999315 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999318 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999320 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999323 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999326 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999329 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999332 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999334 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999337 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999340 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:24.999385 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999342 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999345 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999349 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999360 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999363 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999366 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999369 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999371 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999374 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999377 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999379 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999382 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999385 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999387 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999390 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999394 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999397 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999400 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999403 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:24.999860 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999405 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999408 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999411 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999414 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999417 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999419 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999422 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999425 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999427 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999430 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999433 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999435 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999438 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999440 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999443 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999446 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999449 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999452 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999455 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.000322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999458 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999460 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999463 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999466 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999468 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999471 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999474 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999476 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999479 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999481 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999484 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999489 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999493 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999496 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999499 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999501 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999504 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999507 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999509 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.000757 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999512 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999514 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999518 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999520 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999523 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999526 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999528 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999531 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999535 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999538 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999540 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999543 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999553 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999556 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999559 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999562 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999564 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999567 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999570 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999574 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.001253 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999577 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999580 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999583 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999585 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999588 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999591 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:24.999594 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000390 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000398 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000401 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000405 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000407 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000410 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000413 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000415 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000419 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000421 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000424 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000427 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000429 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.001730 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000432 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000436 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000439 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000441 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000444 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000447 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000450 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000453 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000456 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000458 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000461 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000464 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000466 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000469 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000471 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000474 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000477 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000479 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000482 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000485 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.002200 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000487 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000490 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000493 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000496 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000499 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000501 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000504 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000507 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000509 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000512 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000515 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000517 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000520 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000523 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000526 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000529 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000531 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000535 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000538 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.002681 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000540 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000543 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000545 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000548 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000551 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000554 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000556 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000559 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000561 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000564 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000566 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000569 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000571 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000574 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000577 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000579 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000582 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000584 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000587 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000590 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:25.003127 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000592 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000595 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000598 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000601 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000604 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000606 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000609 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000611 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000614 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000617 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000620 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000622 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000625 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000628 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000631 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000633 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000636 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000638 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000641 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.003620 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000643 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000646 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000649 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000651 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000654 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000656 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000659 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000662 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000666 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000671 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000675 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000679 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000683 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000686 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.000689 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001194 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001202 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001205 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001210 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:25.004057 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001213 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001216 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001219 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001222 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001224 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001227 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001230 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001234 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001236 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001239 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001257 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001262 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001266 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001269 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001272 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001275 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001278 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001280 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001284 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001286 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:25.004519 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001289 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001291 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001297 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001301 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001304 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001307 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001309 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001312 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001315 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001318 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001321 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001323 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001326 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001329 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001331 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001334 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001337 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001339 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.004980 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001342 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001345 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001355 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001360 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001363 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001366 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001369 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001373 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001377 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001381 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001384 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001387 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001390 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001392 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001395 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001397 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001401 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001406 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001409 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:25.005445 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001411 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001414 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001417 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001420 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001423 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001426 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001429 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001431 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001434 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001437 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001440 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001443 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001445 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001448 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001452 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001455 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001458 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001460 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001463 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001466 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.005941 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001468 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001471 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001474 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001477 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001480 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001483 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001485 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001488 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001490 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001493 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001497 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001500 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001502 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001505 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001508 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001510 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001513 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001516 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001518 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:25.006428 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001521 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001524 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001527 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001530 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.001533 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001621 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001629 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001636 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001642 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001647 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001651 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001655 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001659 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001663 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001666 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001670 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001673 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001676 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001680 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001683 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001686 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001689 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001692 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:57:25.006869 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001698 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001701 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001704 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001707 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001711 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001714 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001717 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001721 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001724 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001727 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001730 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001734 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001738 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001741 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001746 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001749 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001752 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001755 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001759 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001762 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001767 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001770 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001773 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001776 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:57:25.007430 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001779 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001784 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001788 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001791 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001794 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001798 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001801 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001804 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001808 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001811 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001815 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001818 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001822 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001825 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001828 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001832 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001835 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001838 2577 flags.go:64] FLAG: --help="false" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001841 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001844 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001848 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001851 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001854 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:57:25.008041 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001858 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001861 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001864 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001867 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001871 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001874 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001877 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001880 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001883 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001886 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001889 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001893 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001896 2577 flags.go:64] FLAG: --lock-file="" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001900 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001903 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001906 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001911 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001916 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001920 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001923 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001926 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001930 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001932 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001935 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001941 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:57:25.008606 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001944 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001948 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001952 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001955 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001958 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001962 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001965 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001968 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001971 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001978 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001981 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001984 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001988 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001991 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001994 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.001997 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002000 2577 flags.go:64] FLAG: --port="10250" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002003 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002008 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0432ec29561a4d6c8" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002011 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002014 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002017 2577 flags.go:64] FLAG: --register-node="true" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002020 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002024 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002028 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002031 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:57:25.009595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002034 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002038 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002041 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002044 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002047 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002050 2577 flags.go:64] FLAG: --runonce="false" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002053 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002057 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002060 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002063 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002066 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002069 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002072 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002076 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002079 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002082 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002085 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002088 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002091 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002095 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002098 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002103 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002106 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002109 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002115 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:57:25.010270 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002118 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002121 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002124 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002127 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002130 2577 flags.go:64] FLAG: --v="2" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002139 2577 flags.go:64] FLAG: --version="false" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002143 2577 flags.go:64] FLAG: --vmodule="" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002147 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002151 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002311 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002316 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002319 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002321 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002324 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002327 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002330 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002332 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002335 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002338 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002340 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002343 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.010857 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002349 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002352 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002354 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002357 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002360 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002363 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002366 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002368 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002371 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002375 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002378 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002381 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002383 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002386 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002389 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002393 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002395 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002398 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002401 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.011377 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002404 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002407 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002409 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002412 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002415 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002418 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002420 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002423 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002426 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002428 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002431 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002434 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002436 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002439 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002442 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002444 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002447 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002450 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002452 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.011845 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002455 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002458 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002460 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002464 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002467 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002470 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002472 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002476 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002480 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002484 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002487 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002489 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002492 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002495 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002498 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002500 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002503 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002506 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002509 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.012375 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002511 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002514 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002517 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002519 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002522 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002525 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002528 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002530 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002533 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002536 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002538 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002543 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002546 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002549 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002553 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002555 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002559 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002562 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002565 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002567 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.012832 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002570 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002572 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002577 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002580 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002582 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002585 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002588 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002591 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002594 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002596 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002599 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002602 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002604 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002607 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002610 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002612 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.013298 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.002638 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002674 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002678 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002681 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002684 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002687 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002690 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002693 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002696 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002698 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002701 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002705 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002707 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002712 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002714 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002717 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002720 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002723 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002726 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002729 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:25.013692 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002732 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002735 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002737 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002740 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002743 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002746 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002748 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002751 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002754 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002756 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002759 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002762 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002765 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002767 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002769 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002772 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002775 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002777 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002780 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.014163 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002783 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002785 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002788 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002790 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002793 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002797 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002801 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002804 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002806 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002809 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002811 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002814 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002817 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002820 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002822 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002825 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002827 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002830 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002834 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.014622 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002838 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002842 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002845 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002847 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002850 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002853 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002855 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002858 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002861 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002863 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002866 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002869 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002871 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002874 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002876 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002879 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002882 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002885 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002887 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002894 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:25.015097 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002897 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002900 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002904 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002907 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002910 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002913 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002916 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002918 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002921 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002924 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002926 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002929 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002932 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002935 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002937 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002940 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002943 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002946 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002949 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002951 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:25.015581 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002954 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002956 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002959 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002962 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002964 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002967 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002971 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002973 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.002976 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.010270 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.010287 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.011576 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.011583 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011644 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011650 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011654 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011658 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011661 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011664 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.016040 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011667 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011670 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011672 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011675 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011678 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011681 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011684 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011687 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011690 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011693 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011696 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011699 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011702 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011704 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011708 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011712 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011716 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011719 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011722 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011725 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:25.016544 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011728 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011730 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011734 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011737 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011739 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011742 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011745 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011748 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011750 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011753 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011758 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011766 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011769 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011773 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011776 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011778 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011781 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011784 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.017018 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011786 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011790 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011793 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011796 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011799 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011802 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011804 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011807 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011810 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011813 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011815 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011818 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011821 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011823 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011826 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011829 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011832 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011835 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011838 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.017507 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011841 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011844 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011847 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011850 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011853 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011856 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011859 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011862 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011864 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011867 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011870 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011873 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011876 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011878 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011881 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011884 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011887 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011890 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011892 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011895 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.017950 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011898 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011901 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011903 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011906 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011908 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011911 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011914 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011916 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011919 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011923 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011925 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011928 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011931 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011934 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011937 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011939 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011942 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011945 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011947 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011950 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.018429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011953 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.011955 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.011986 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012117 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012122 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012125 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012128 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012131 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012134 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012137 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012140 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012143 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012145 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012149 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012153 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.018907 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012156 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012159 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012162 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012165 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012167 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012170 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012173 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012176 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012178 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012181 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012184 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012187 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012189 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012192 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012195 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012197 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012200 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012202 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012205 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012207 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.019284 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012210 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012213 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012215 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012218 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012221 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012223 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012227 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012229 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012232 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012234 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012237 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012240 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012259 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012264 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012268 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012271 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012273 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012276 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012279 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:25.019906 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012282 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012284 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012287 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012290 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012293 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012295 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012298 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012300 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012303 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012306 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012308 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012311 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012314 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012316 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012319 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012322 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012324 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012327 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012329 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012332 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:25.020365 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012336 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012339 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012342 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012344 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012347 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012349 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012352 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012355 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012357 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012359 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012362 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012366 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012368 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012371 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012374 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012376 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012379 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012382 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012385 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.020837 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012388 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012391 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012394 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012396 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012399 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012401 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012404 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012406 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012409 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012412 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012414 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012417 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012419 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012422 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012425 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012428 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.021289 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.012451 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012483 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012487 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012490 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012493 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012496 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012498 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012502 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012504 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012507 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012510 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012512 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012515 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012517 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012520 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012523 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012525 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012528 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012531 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:25.021696 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012534 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012536 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012539 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012541 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012544 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012547 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012549 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012552 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012555 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012558 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012561 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012563 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012566 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012569 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012572 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012574 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012577 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012580 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012582 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012585 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:25.022131 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012588 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012591 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012593 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012596 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012598 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012601 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012604 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012606 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012609 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012611 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012614 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012616 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012619 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012622 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012624 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012627 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012629 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012632 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012634 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012638 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:25.022611 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012640 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012644 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012648 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012651 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012654 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012658 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012660 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012663 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012666 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012669 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012672 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012674 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012677 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012680 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012682 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012685 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012687 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012690 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012692 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:25.023090 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012695 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012697 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012700 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012702 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012705 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012708 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012710 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012713 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012716 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012718 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012721 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012723 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012726 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012728 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012735 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012738 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012742 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012746 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012749 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012752 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:25.023536 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012755 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012758 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012760 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012763 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012766 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012769 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012772 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012794 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:25.012798 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.012853 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.014740 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.015680 2577 server.go:1014] "Starting client certificate rotation" Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.015772 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.015807 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.017857 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:25.023988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.018924 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:25.030990 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.030970 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:57:25.034557 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.034541 2577 log.go:25] "Validated CRI v1 image API" Oct 30 17:57:25.035383 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.035358 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:57:25.039027 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.038997 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 873a89f6-9599-4b6d-96a7-48069d8ff286:/dev/nvme0n1p3 957d7702-5b85-4489-86b0-843ba1e18de3:/dev/nvme0n1p4] Oct 30 17:57:25.039115 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.039025 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:57:25.044026 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.043905 2577 manager.go:217] Machine: {Timestamp:2026-10-30 17:57:25.042602563 +0000 UTC m=+0.330712288 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097439 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b3a7548125186f5d9763963d78dfb SystemUUID:ec2b3a75-4812-5186-f5d9-763963d78dfb BootID:7c3d372c-ac0e-4d9e-8931-432f273512d9 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fd:e8:9f:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fd:e8:9f:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:20:5e:39:f0:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:57:25.044600 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.044587 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:57:25.044717 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.044696 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:57:25.047776 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.047744 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:57:25.047922 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.047776 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:57:25.047975 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.047924 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:57:25.047975 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.047932 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:57:25.047975 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.047968 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:25.048528 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.048518 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:25.049314 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.049305 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:57:25.051968 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.051957 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:57:25.052088 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.052079 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:57:25.052803 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.052793 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:57:25.052835 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.052808 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:57:25.052835 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.052817 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:57:25.052835 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.052826 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:57:25.052835 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.052834 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:57:25.053182 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.053171 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:25.055505 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.055486 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:57:25.060441 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.060429 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:25.061535 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061523 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:57:25.061581 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061553 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:57:25.061705 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061694 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:57:25.061742 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061710 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:57:25.061742 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061716 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:57:25.061742 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061722 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:57:25.061742 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061728 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:57:25.061742 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.061734 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:57:25.062282 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062273 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:57:25.062316 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062282 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:57:25.062316 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062290 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:57:25.062316 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062296 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:57:25.062316 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062305 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:57:25.062316 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062313 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:57:25.062883 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062875 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:57:25.062883 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.062883 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:57:25.066506 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.066492 2577 server.go:1263] "Started kubelet" Oct 30 17:57:25.066595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.066569 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:57:25.066654 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.066614 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:57:25.066684 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.066672 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:57:25.067261 ip-10-0-133-124 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:57:25.067856 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.067833 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:57:25.068512 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.068497 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:57:25.073531 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.073503 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:57:25.074715 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.074703 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:57:25.074768 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.074727 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:57:25.074814 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.074792 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:25.075273 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075237 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:57:25.075352 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075254 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:57:25.075438 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075358 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:57:25.075484 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075447 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:57:25.075484 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075457 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:57:25.075737 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075719 2577 factory.go:55] Registering systemd factory Oct 30 17:57:25.075804 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075740 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:57:25.075963 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075942 2577 factory.go:153] Registering CRI-O factory Oct 30 17:57:25.075963 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.075962 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 17:57:25.076097 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.076005 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:57:25.076097 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.076023 2577 factory.go:103] Registering Raw factory Oct 30 17:57:25.076097 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.076036 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 17:57:25.076881 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.076870 2577 manager.go:319] Starting recovery of all containers Oct 30 17:57:25.088399 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.088377 2577 manager.go:324] Recovery completed Oct 30 17:57:25.090227 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.089737 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 17:57:25.092733 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.092718 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.094636 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.094615 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.094688 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.094649 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.094688 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.094658 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.095121 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.095111 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:57:25.095121 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.095116 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.095121 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.095120 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:57:25.095217 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.095135 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:57:25.095849 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.095828 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.095920 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.095863 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.095920 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.095878 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.101736 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.101711 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:25.101736 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.101712 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:57:25.102689 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.101714 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b3651d89 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.066464649 +0000 UTC m=+0.354574369,LastTimestamp:2026-10-30 17:57:25.066464649 +0000 UTC m=+0.354574369,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.120393 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.120362 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:57:25.120479 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.120326 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.120479 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.120377 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:57:25.120600 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.120580 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:57:25.141850 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.141769 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5133dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,LastTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.172901 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5136027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,LastTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.175819 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.215506 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.216650 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.216677 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.216713 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.216762 2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.224534 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b512fa70\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.095845943 +0000 UTC m=+0.383955664,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.233839 2577 policy_none.go:50] "Start" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.233854 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:57:25.234529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.233875 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:57:25.237374 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.235552 2577 policy_none.go:44] "Start" Oct 30 17:57:25.266927 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.266763 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:57:25.271368 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.271353 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 17:57:25.271455 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.271385 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:57:25.271455 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.271397 2577 server.go:87] "Starting device plugin registration server" Oct 30 17:57:25.271547 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.270983 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5133dd5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5133dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,LastTimestamp:2026-10-30 17:57:25.095870585 +0000 UTC m=+0.383980305,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.271667 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.271655 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:57:25.271724 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.271668 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:57:25.271808 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.271792 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:57:25.271888 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.271863 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:57:25.271888 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.271874 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:57:25.272388 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.272362 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:57:25.272477 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.272410 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:25.317210 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.317187 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal"] Oct 30 17:57:25.317292 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.317260 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.317933 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.317914 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.318002 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.317940 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.318002 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.317949 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.319959 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.319931 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.320123 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320108 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.320180 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320170 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.320622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320606 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.320696 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320631 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.320696 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320645 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.320696 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320647 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.320696 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320664 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.320696 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.320688 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.320960 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.320934 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:57:25.321041 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.320984 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Oct 30 17:57:25.321079 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.320984 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5136027\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5136027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,LastTimestamp:2026-10-30 17:57:25.095882016 +0000 UTC m=+0.383991736,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.322755 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.322738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.322846 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.322767 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.323440 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.323425 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.323529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.323450 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.323529 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.323462 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.345781 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.345714 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060bf7d2404 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.269365764 +0000 UTC m=+0.557475475,LastTimestamp:2026-10-30 17:57:25.269365764 +0000 UTC m=+0.557475475,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.347436 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.347419 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.350110 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.350094 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.369569 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.369509 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060bf9b233b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.271331643 +0000 UTC m=+0.559441350,LastTimestamp:2026-10-30 17:57:25.271331643 +0000 UTC m=+0.559441350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.372190 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.372176 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.372779 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.372764 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.372853 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.372786 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.372853 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.372795 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.372853 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.372813 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.393501 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.393424 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b512fa70\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.317929 +0000 UTC m=+0.606038720,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.410835 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.410814 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.410901 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.410818 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5133dd5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5133dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,LastTimestamp:2026-10-30 17:57:25.317944432 +0000 UTC m=+0.606054151,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.436854 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.436793 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5136027\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5136027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,LastTimestamp:2026-10-30 17:57:25.317953318 +0000 UTC m=+0.606063039,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.452337 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.452282 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b512fa70\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.320621453 +0000 UTC m=+0.608731176,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.476687 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.476669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c67125f85ae263b41b821237802c161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" (UID: \"2c67125f85ae263b41b821237802c161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.476767 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.476690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.476767 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.476708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.482442 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.482390 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5133dd5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5133dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,LastTimestamp:2026-10-30 17:57:25.320637775 +0000 UTC m=+0.608747497,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.525672 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.525564 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5136027\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5136027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,LastTimestamp:2026-10-30 17:57:25.320650568 +0000 UTC m=+0.608760295,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.568919 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.568854 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b512fa70\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.320656601 +0000 UTC m=+0.608766325,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.577644 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.577625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c67125f85ae263b41b821237802c161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" (UID: \"2c67125f85ae263b41b821237802c161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.577700 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.577649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.577700 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.577665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.577794 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.577714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.577794 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.577726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c67125f85ae263b41b821237802c161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" (UID: \"2c67125f85ae263b41b821237802c161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.577794 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.577747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.610836 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.610770 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5133dd5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5133dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,LastTimestamp:2026-10-30 17:57:25.320669576 +0000 UTC m=+0.608779297,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.610995 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.610981 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:25.612134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.612117 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:25.612202 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.612145 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:25.612202 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.612155 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:25.612202 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.612184 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.649965 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.649947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.651926 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.651872 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5136027\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5136027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,LastTimestamp:2026-10-30 17:57:25.32069458 +0000 UTC m=+0.608804303,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.654342 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:25.654332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.693913 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.693890 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:25.693998 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.693917 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b512fa70\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.323437758 +0000 UTC m=+0.611547478,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.738507 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.738485 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Oct 30 17:57:25.738573 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.738476 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5133dd5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5133dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,LastTimestamp:2026-10-30 17:57:25.323456762 +0000 UTC m=+0.611566482,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.771613 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.771553 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5136027\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5136027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,LastTimestamp:2026-10-30 17:57:25.323465545 +0000 UTC m=+0.611575265,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.813490 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.813389 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b512fa70\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.372775663 +0000 UTC m=+0.660885384,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.855849 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.855778 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5133dd5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5133dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094653397 +0000 UTC m=+0.382763117,LastTimestamp:2026-10-30 17:57:25.372790509 +0000 UTC m=+0.660900229,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.898028 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.897967 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b5136027\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b5136027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094662183 +0000 UTC m=+0.382771903,LastTimestamp:2026-10-30 17:57:25.372799093 +0000 UTC m=+0.660908813,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:25.935254 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:25.935179 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-124.ec2.internal.18e36060b512fa70\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-124.ec2.internal.18e36060b512fa70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-124.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:25.094636144 +0000 UTC m=+0.382745864,LastTimestamp:2026-10-30 17:57:25.612132266 +0000 UTC m=+0.900241988,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:27.037780 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.037745 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:57:27.097392 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.097340 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:27.099167 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.099148 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:27.099281 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.099179 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:27.099281 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.099188 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:27.099281 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.099216 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:27.136751 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.136731 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:27.137844 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.137813 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:27.137951 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.137863 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:27.137951 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.137877 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:27.144699 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.144678 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:27.190164 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.190135 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:27.245172 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:27.245142 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9108d1d7bcbaabf4aba6b0607ef8c386.slice/crio-e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16 WatchSource:0}: Error finding container e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16: Status 404 returned error can't find the container with id e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16 Oct 30 17:57:27.245535 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:27.245509 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c67125f85ae263b41b821237802c161.slice/crio-a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a WatchSource:0}: Error finding container a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a: Status 404 returned error can't find the container with id a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a Oct 30 17:57:27.250156 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.250138 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:57:27.301910 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.301833 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal.18e36061359caa78 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal,UID:9108d1d7bcbaabf4aba6b0607ef8c386,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:27.251143288 +0000 UTC m=+1.536678054,LastTimestamp:2026-10-30 17:57:27.251143288 +0000 UTC m=+1.536678054,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:27.347061 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.346983 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-124.ec2.internal.18e3606135c42fcb kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-124.ec2.internal,UID:2c67125f85ae263b41b821237802c161,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:27.253733323 +0000 UTC m=+1.539268083,LastTimestamp:2026-10-30 17:57:27.253733323 +0000 UTC m=+1.539268083,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:27.440921 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.440854 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:57:27.440921 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.440854 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:57:27.583560 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.583531 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Oct 30 17:57:27.624078 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:27.624046 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:57:27.990751 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.990705 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:27.992169 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.992145 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:27.992330 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.992189 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:27.992330 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.992205 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:27.992330 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:27.992240 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:28.086112 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:28.086074 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:28.146190 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:28.146159 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:28.224591 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:28.224492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerStarted","Data":"e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16"} Oct 30 17:57:28.225718 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:28.225694 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" event={"ID":"2c67125f85ae263b41b821237802c161","Type":"ContainerStarted","Data":"a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a"} Oct 30 17:57:28.493588 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:28.493472 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal.18e360617ce2ecac openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal,UID:9108d1d7bcbaabf4aba6b0607ef8c386,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 1.195s (1.195s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:28.446930092 +0000 UTC m=+2.732464863,LastTimestamp:2026-10-30 17:57:28.446930092 +0000 UTC m=+2.732464863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:28.717791 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:28.717654 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal.18e360618a2a0dab openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal,UID:9108d1d7bcbaabf4aba6b0607ef8c386,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:28.669695403 +0000 UTC m=+2.955230150,LastTimestamp:2026-10-30 17:57:28.669695403 +0000 UTC m=+2.955230150,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:28.762463 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:28.762369 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal.18e360618ac4228b openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal,UID:9108d1d7bcbaabf4aba6b0607ef8c386,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:28.679793291 +0000 UTC m=+2.965328052,LastTimestamp:2026-10-30 17:57:28.679793291 +0000 UTC m=+2.965328052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:28.901150 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:28.901060 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-124.ec2.internal.18e3606195616199 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-124.ec2.internal,UID:2c67125f85ae263b41b821237802c161,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 1.604s (1.604s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:28.857870745 +0000 UTC m=+3.143405515,LastTimestamp:2026-10-30 17:57:28.857870745 +0000 UTC m=+3.143405515,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:28.952340 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:28.952262 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-124.ec2.internal.18e36061980511b4 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-124.ec2.internal,UID:2c67125f85ae263b41b821237802c161,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:28.902152628 +0000 UTC m=+3.187687374,LastTimestamp:2026-10-30 17:57:28.902152628 +0000 UTC m=+3.187687374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:28.995525 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:28.995412 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-124.ec2.internal.18e36061987115a7 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-124.ec2.internal,UID:2c67125f85ae263b41b821237802c161,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-133-124.ec2.internal,},FirstTimestamp:2026-10-30 17:57:28.909231527 +0000 UTC m=+3.194766288,LastTimestamp:2026-10-30 17:57:28.909231527 +0000 UTC m=+3.194766288,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-124.ec2.internal,}" Oct 30 17:57:29.153879 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.153852 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:29.228596 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.228560 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerStarted","Data":"f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261"} Oct 30 17:57:29.228596 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.228596 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:29.229397 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.229380 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:29.229502 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.229410 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:29.229502 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.229420 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:29.229612 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:29.229497 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Oct 30 17:57:29.229612 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:29.229595 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:29.230083 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.230062 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" event={"ID":"2c67125f85ae263b41b821237802c161","Type":"ContainerStarted","Data":"dbd3a9173fe1dc01879b9834e9a35829fc8644635f4e8b3cce3f9b80a92233a0"} Oct 30 17:57:29.230138 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.230098 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:29.230748 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.230735 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:29.230802 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.230755 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:29.230802 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.230768 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:29.230942 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:29.230930 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:29.356621 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.356570 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:29.357448 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.357431 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:29.357516 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.357460 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:29.357516 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.357471 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:29.389177 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:29.389151 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:57:29.686703 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.686618 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:29.687584 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.687563 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:29.687689 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.687603 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:29.687689 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.687616 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:29.687689 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:29.687648 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:29.779652 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:29.779620 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:30.144035 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.143969 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:30.231733 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.231712 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:30.232225 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.231717 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:30.232466 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.232449 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:30.232544 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.232482 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:30.232544 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.232498 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:30.232544 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.232537 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:30.232684 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.232562 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:30.232684 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:30.232573 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:30.232807 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:30.232793 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:30.232969 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:30.232948 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:30.297012 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:30.296976 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:57:30.297012 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:30.296987 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:57:30.854712 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:30.854637 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:57:31.161800 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:31.161769 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:32.153017 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:32.152986 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:32.980182 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:32.980145 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:32.981254 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:32.981236 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:32.981346 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:32.981290 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:32.981346 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:32.981302 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:32.981346 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:32.981329 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:33.027044 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:33.027020 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Oct 30 17:57:33.068651 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:33.068625 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:33.143834 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:33.143800 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:33.530002 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:33.529970 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:57:34.021999 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:34.021922 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:34.022965 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:34.022946 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:34.023067 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:34.022978 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:34.023067 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:34.022989 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:34.152204 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:34.152172 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:34.608522 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:34.608484 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:57:35.160882 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:35.160845 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:57:35.160882 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:35.160870 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:35.209717 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:35.209690 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:57:36.107745 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.107711 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:57:36.120870 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.120846 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhfjh" Oct 30 17:57:36.128286 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.128266 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhfjh" Oct 30 17:57:36.163649 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.163629 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.189951 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.189937 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.255945 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.255919 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.275087 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:36.275064 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:36.528415 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.528336 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.528415 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:36.528363 2577 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.576883 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.576859 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.603107 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.603088 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.673325 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.673298 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.950047 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:36.949988 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:36.950047 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:36.950012 2577 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:37.017724 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:37.017700 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:57:37.131735 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:37.131690 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:52:36 +0000 UTC" deadline="2028-07-10 07:59:35.106937834 +0000 UTC" Oct 30 17:57:37.131735 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:37.131728 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14846h1m57.97521383s" Oct 30 17:57:37.263495 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:37.263430 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:37.290382 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:37.290361 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:37.361483 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:37.361465 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:37.632032 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:37.632010 2577 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:37.632032 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:37.632035 2577 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-133-124.ec2.internal" not found Oct 30 17:57:39.460611 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:39.460427 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:39.469535 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:39.469512 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:39.471428 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:39.471411 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:39.471539 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:39.471442 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:39.471539 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:39.471457 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:39.471539 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:39.471491 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:39.495376 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:39.495355 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:39.495477 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:39.495380 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-124.ec2.internal\": node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:39.543757 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:39.543737 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:39.644248 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:39.644219 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:39.745288 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:39.745239 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:39.845755 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:39.845713 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:39.946181 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:39.946152 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.046350 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.046261 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.147200 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.147152 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.245166 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:40.245144 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal_9108d1d7bcbaabf4aba6b0607ef8c386/setup/0.log" Oct 30 17:57:40.245291 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:40.245177 2577 generic.go:356] "Generic (PLEG): container finished" podID="9108d1d7bcbaabf4aba6b0607ef8c386" containerID="f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261" exitCode=1 Oct 30 17:57:40.245291 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:40.245204 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerDied","Data":"f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261"} Oct 30 17:57:40.245354 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:40.245324 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:40.246282 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:40.246267 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:40.246377 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:40.246296 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:40.246377 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:40.246306 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:40.246507 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.246493 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:40.247556 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.247542 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.348253 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.348232 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.448735 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.448710 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.549347 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.549323 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.650000 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.649977 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.750456 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.750425 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.850992 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.850970 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:40.952001 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:40.951946 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.052957 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.052926 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.153723 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.153705 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.247964 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:41.247906 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal_9108d1d7bcbaabf4aba6b0607ef8c386/setup/0.log" Oct 30 17:57:41.247964 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:41.247948 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerStarted","Data":"2a00aa2391414eb1b75d1d9a6a793eae385d2d7f3a6912082486ed8c8783f2c7"} Oct 30 17:57:41.248108 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:41.248094 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:41.252647 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:41.252631 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:41.252727 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:41.252659 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:41.252727 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:41.252669 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:41.252856 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.252844 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-124.ec2.internal\" not found" node="ip-10-0-133-124.ec2.internal" Oct 30 17:57:41.253861 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.253848 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.354337 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.354317 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.455285 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.455265 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.555377 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.555312 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.656296 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.656278 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.757212 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.757186 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.857693 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.857651 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:41.958140 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:41.958110 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.058833 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.058796 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.158905 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.158871 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.259413 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.259390 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.360082 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.360049 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.461169 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.461093 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.561209 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.561179 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.662068 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.662034 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.762541 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.762490 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.863076 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.863051 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:42.963538 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:42.963508 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.064355 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.064300 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.165123 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.165096 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.265684 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.265663 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.366320 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.366264 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.466868 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.466808 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.566974 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.566945 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.606350 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:43.606325 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:43.607510 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:43.607492 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:43.607573 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:43.607529 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:43.607573 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:43.607544 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:43.667912 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.667879 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.768369 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.768330 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.868891 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.868873 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:43.969373 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:43.969327 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:44.070220 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:44.070195 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:44.170791 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:44.170762 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:57:44.260724 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:44.260673 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:44.277763 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:44.277743 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:57:44.303029 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:44.303013 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:44.304641 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:44.304629 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:57:44.326197 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:44.326178 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:45.067190 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.067154 2577 apiserver.go:52] "Watching apiserver" Oct 30 17:57:45.089453 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.089431 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:57:45.113482 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.113458 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:57:45.113769 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.113754 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-jrtkt","openshift-multus/multus-ckrtw","openshift-ovn-kubernetes/ovnkube-node-rb89h","kube-system/global-pull-secret-syncer-jg28z","kube-system/konnectivity-agent-5jtxd","kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal","openshift-image-registry/node-ca-rfjbn","openshift-multus/multus-additional-cni-plugins-86gbr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal","openshift-multus/network-metrics-daemon-prjhj","openshift-network-diagnostics/network-check-target-c5qcf","openshift-network-operator/iptables-alerter-c5z8z","openshift-cluster-node-tuning-operator/tuned-d65dr"] Oct 30 17:57:45.119362 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.119346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.122450 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.122428 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.125608 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.125593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.128587 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.128569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.128660 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.128643 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:45.131539 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.131523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.134428 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.134412 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.136497 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.136481 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzq26\"" Oct 30 17:57:45.140216 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.137927 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:57:45.140216 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.138031 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.140216 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.138267 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.140216 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.138382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.140216 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.140130 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.140216 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.140160 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.140537 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.140522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.144362 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.144343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.145534 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.145517 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:57:45.145605 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.145573 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6627\"" Oct 30 17:57:45.146637 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.146619 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:57:45.146697 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.146683 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vrf2s" Oct 30 17:57:45.147422 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.147398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.150747 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.150729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:45.150854 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.150780 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:45.153698 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.153681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:45.153781 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.153741 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:45.178910 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.178893 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:57:45.179010 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.178996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-modprobe-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.179044 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-bin\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179044 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-netd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179127 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-device-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.179127 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-k8s-cni-cncf-io\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179196 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-daemon-config\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179196 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-multus-certs\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179196 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cnibin\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.179326 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179326 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179326 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-conf-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179466 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9fw\" (UniqueName: \"kubernetes.io/projected/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-kube-api-access-vp9fw\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.179466 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-systemd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179466 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-log-socket\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179466 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-konnectivity-ca\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.179466 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-var-lib-kubelet\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.179624 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3952ef0-db1a-4c93-b9e4-fccda9618beb-host-slash\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.179624 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-kubelet\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179624 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-socket-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.179624 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cni-binary-copy\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179624 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-socket-dir-parent\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179624 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-netns\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179624 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-host\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-conf\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-tmp\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3952ef0-db1a-4c93-b9e4-fccda9618beb-iptables-alerter-script\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xd4\" (UniqueName: \"kubernetes.io/projected/e3952ef0-db1a-4c93-b9e4-fccda9618beb-kube-api-access-r8xd4\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-ovn\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-script-lib\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179814 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-systemd-units\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-multus\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.179923 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bd65\" (UniqueName: \"kubernetes.io/projected/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-kube-api-access-2bd65\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-system-cni-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.179990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-host\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-tuned\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-registration-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4jt\" (UniqueName: \"kubernetes.io/projected/b52cb20e-db43-407c-bf36-870475ceadb7-kube-api-access-8t4jt\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnxb\" (UniqueName: \"kubernetes.io/projected/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-kube-api-access-ncnxb\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-slash\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-config\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chtw\" (UniqueName: \"kubernetes.io/projected/a2daed88-3483-4e79-a6cc-f12cddc1bdac-kube-api-access-9chtw\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hbk\" (UniqueName: \"kubernetes.io/projected/2cf30b64-3606-4e62-894c-1c5fbb2e7190-kube-api-access-g6hbk\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:45.180285 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-agent-certs\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-etc-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovn-node-metrics-cert\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7d3975-f96e-4f38-9535-1f37d2142493-hosts-file\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cnibin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-kubelet\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-hostroot\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-serviceca\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-dbus\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-systemd\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7d3975-f96e-4f38-9535-1f37d2142493-tmp-dir\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-system-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-bin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-etc-kubernetes\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-kubernetes\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.180935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-run\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmj9k\" (UniqueName: \"kubernetes.io/projected/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-kube-api-access-tmj9k\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysconfig\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-sys\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-lib-modules\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-var-lib-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-sys-fs\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.180997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-netns\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.181026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-node-log\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.181051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-env-overrides\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.181073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-etc-selinux\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.181092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9l7f\" (UniqueName: \"kubernetes.io/projected/6f7d3975-f96e-4f38-9535-1f37d2142493-kube-api-access-h9l7f\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.181113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.181472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.181136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-kubelet-config\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.237007 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.236982 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:57:45.238256 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238240 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:57:45.238354 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238269 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:57:45.238354 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238289 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh48h\"" Oct 30 17:57:45.238354 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238304 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.238354 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238322 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.238354 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238332 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.238584 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238361 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.238584 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238401 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:57:45.238584 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238528 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:57:45.238584 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238544 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.238733 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238649 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.238872 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238853 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:57:45.238872 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238864 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:57:45.238969 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238858 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p48xz\"" Oct 30 17:57:45.238969 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.238886 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:57:45.242405 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.242394 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt2qr\"" Oct 30 17:57:45.242467 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.242407 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wj5nd\"" Oct 30 17:57:45.242467 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.242442 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.251183 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.251172 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:57:45.252194 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252183 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.252262 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252200 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nwsqb\"" Oct 30 17:57:45.252262 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252201 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:57:45.252262 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252217 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:57:45.252526 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252514 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9m88r\"" Oct 30 17:57:45.252526 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252525 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:57:45.252631 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252548 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jcq9r\"" Oct 30 17:57:45.252631 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.252623 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:57:45.282246 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chtw\" (UniqueName: \"kubernetes.io/projected/a2daed88-3483-4e79-a6cc-f12cddc1bdac-kube-api-access-9chtw\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.282335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.282335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hbk\" (UniqueName: \"kubernetes.io/projected/2cf30b64-3606-4e62-894c-1c5fbb2e7190-kube-api-access-g6hbk\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:45.282335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-agent-certs\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.282335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-etc-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.282335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:45.282335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovn-node-metrics-cert\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7d3975-f96e-4f38-9535-1f37d2142493-hosts-file\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cnibin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282383 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-etc-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-kubelet\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-hostroot\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-serviceca\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282467 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7d3975-f96e-4f38-9535-1f37d2142493-hosts-file\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-dbus\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-systemd\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282520 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-kubelet\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7d3975-f96e-4f38-9535-1f37d2142493-tmp-dir\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-system-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-bin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282579 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-hostroot\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.282622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-etc-kubernetes\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-kubernetes\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-run\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmj9k\" (UniqueName: \"kubernetes.io/projected/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-kube-api-access-tmj9k\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282718 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-systemd\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-dbus\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysconfig\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282774 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-etc-kubernetes\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-sys\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-lib-modules\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282867 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-kubernetes\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282855 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282886 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-system-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysconfig\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282576 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cnibin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282969 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-var-lib-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.283237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-var-lib-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.282986 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282994 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-sys\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.282939 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-bin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-sys-fs\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-netns\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-netns\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.283129 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:45.783077056 +0000 UTC m=+20.068611816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283147 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-lib-modules\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-sys-fs\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-run\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-node-log\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283212 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-node-log\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-env-overrides\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-etc-selinux\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9l7f\" (UniqueName: \"kubernetes.io/projected/6f7d3975-f96e-4f38-9535-1f37d2142493-kube-api-access-h9l7f\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.283993 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-kubelet-config\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-modprobe-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-bin\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283377 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-etc-selinux\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-netd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-device-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-k8s-cni-cncf-io\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283499 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-kubelet-config\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283526 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-netd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-daemon-config\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283546 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283500 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7d3975-f96e-4f38-9535-1f37d2142493-tmp-dir\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-multus-certs\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cnibin\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-modprobe-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.284673 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283658 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-k8s-cni-cncf-io\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-conf-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283710 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-device-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283721 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cnibin\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9fw\" (UniqueName: \"kubernetes.io/projected/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-kube-api-access-vp9fw\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283670 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-multus-certs\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283742 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-conf-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-systemd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283892 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-systemd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-log-socket\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283949 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-serviceca\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.283963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-konnectivity-ca\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284005 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-log-socket\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284011 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284013 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-bin\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-var-lib-kubelet\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.285445 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3952ef0-db1a-4c93-b9e4-fccda9618beb-host-slash\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284174 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-var-lib-kubelet\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284187 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-daemon-config\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-kubelet\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284245 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-kubelet\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3952ef0-db1a-4c93-b9e4-fccda9618beb-host-slash\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-socket-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cni-binary-copy\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-socket-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-socket-dir-parent\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-netns\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-host\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-conf\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-netns\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-tmp\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-socket-dir-parent\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.286253 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-host\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284781 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-conf\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3952ef0-db1a-4c93-b9e4-fccda9618beb-iptables-alerter-script\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xd4\" (UniqueName: \"kubernetes.io/projected/e3952ef0-db1a-4c93-b9e4-fccda9618beb-kube-api-access-r8xd4\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-ovn\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-script-lib\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-systemd-units\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.284986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-multus\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bd65\" (UniqueName: \"kubernetes.io/projected/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-kube-api-access-2bd65\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-system-cni-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-host\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-tuned\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285305 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-ovn\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-registration-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.287134 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4jt\" (UniqueName: \"kubernetes.io/projected/b52cb20e-db43-407c-bf36-870475ceadb7-kube-api-access-8t4jt\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285387 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-system-cni-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnxb\" (UniqueName: \"kubernetes.io/projected/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-kube-api-access-ncnxb\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285453 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-systemd-units\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-slash\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285502 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285556 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-multus\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-config\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.285672 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-script-lib\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.285729 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret podName:845bfa8a-e757-4cd4-bcd0-f3625bb21afa nodeName:}" failed. No retries permitted until 2026-10-30 17:57:45.785709269 +0000 UTC m=+20.071244016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret") pod "global-pull-secret-syncer-jg28z" (UID: "845bfa8a-e757-4cd4-bcd0-f3625bb21afa") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285739 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285785 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-host\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285849 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3952ef0-db1a-4c93-b9e4-fccda9618beb-iptables-alerter-script\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.285968 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.287657 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286238 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-env-overrides\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286297 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-slash\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286325 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-config\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286379 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-registration-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286479 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-konnectivity-ca\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cni-binary-copy\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.286802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.287592 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovn-node-metrics-cert\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.288129 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.287668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-agent-certs\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.288570 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.288552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-tmp\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.288982 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.288959 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-tuned\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.309310 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.309212 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmj9k\" (UniqueName: \"kubernetes.io/projected/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-kube-api-access-tmj9k\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.309405 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.309384 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chtw\" (UniqueName: \"kubernetes.io/projected/a2daed88-3483-4e79-a6cc-f12cddc1bdac-kube-api-access-9chtw\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.309925 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.309795 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:45.309925 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.309831 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:45.309925 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.309845 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nbjs2 for pod openshift-network-diagnostics/network-check-target-c5qcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:45.309925 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.309895 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2 podName:bd61c76f-9514-46db-99e2-21c6280a4842 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:45.809880001 +0000 UTC m=+20.095414761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjs2" (UniqueName: "kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2") pod "network-check-target-c5qcf" (UID: "bd61c76f-9514-46db-99e2-21c6280a4842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:45.311322 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.311297 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnxb\" (UniqueName: \"kubernetes.io/projected/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-kube-api-access-ncnxb\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.311595 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.311568 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hbk\" (UniqueName: \"kubernetes.io/projected/2cf30b64-3606-4e62-894c-1c5fbb2e7190-kube-api-access-g6hbk\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:45.311884 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.311852 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bd65\" (UniqueName: \"kubernetes.io/projected/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-kube-api-access-2bd65\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.311884 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.311875 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4jt\" (UniqueName: \"kubernetes.io/projected/b52cb20e-db43-407c-bf36-870475ceadb7-kube-api-access-8t4jt\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.311980 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.311950 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xd4\" (UniqueName: \"kubernetes.io/projected/e3952ef0-db1a-4c93-b9e4-fccda9618beb-kube-api-access-r8xd4\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.312663 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.312647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9l7f\" (UniqueName: \"kubernetes.io/projected/6f7d3975-f96e-4f38-9535-1f37d2142493-kube-api-access-h9l7f\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.313569 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.313552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9fw\" (UniqueName: \"kubernetes.io/projected/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-kube-api-access-vp9fw\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.417988 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.417961 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:45.431101 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.431081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:57:45.437726 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.437711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:57:45.438983 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.438964 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b785d_9cd4_48ed_a4f7_85b5ddcd086c.slice/crio-71ddb07e0b5f77310b67a3c582e4acff6545c9908588f9e1acffae511a57a597 WatchSource:0}: Error finding container 71ddb07e0b5f77310b67a3c582e4acff6545c9908588f9e1acffae511a57a597: Status 404 returned error can't find the container with id 71ddb07e0b5f77310b67a3c582e4acff6545c9908588f9e1acffae511a57a597 Oct 30 17:57:45.445457 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.445435 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3952ef0_db1a_4c93_b9e4_fccda9618beb.slice/crio-1298430fd8f01d571a0d5ea01f613516e727d85850b746bc88a1349dc62c2e86 WatchSource:0}: Error finding container 1298430fd8f01d571a0d5ea01f613516e727d85850b746bc88a1349dc62c2e86: Status 404 returned error can't find the container with id 1298430fd8f01d571a0d5ea01f613516e727d85850b746bc88a1349dc62c2e86 Oct 30 17:57:45.455462 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.455443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:45.457586 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.457569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:57:45.464722 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.464705 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c5c437_9fa5_4d3a_8ba5_ed7b6ff6656a.slice/crio-347508d719c82b6df9a48faca4f859f6fe9dfa4279d07c6f6974c774567c7ea9 WatchSource:0}: Error finding container 347508d719c82b6df9a48faca4f859f6fe9dfa4279d07c6f6974c774567c7ea9: Status 404 returned error can't find the container with id 347508d719c82b6df9a48faca4f859f6fe9dfa4279d07c6f6974c774567c7ea9 Oct 30 17:57:45.466704 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.466688 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd334a8_7dd5_4a51_8ffc_adb0c815682f.slice/crio-b86c58491273d3225d53df5f0c2da6585fac542d2a64281a41773875a5984cfe WatchSource:0}: Error finding container b86c58491273d3225d53df5f0c2da6585fac542d2a64281a41773875a5984cfe: Status 404 returned error can't find the container with id b86c58491273d3225d53df5f0c2da6585fac542d2a64281a41773875a5984cfe Oct 30 17:57:45.468076 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.468061 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:57:45.470812 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.470796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:57:45.472235 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.472222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:57:45.472408 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.472371 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" podStartSLOduration=1.472361563 podStartE2EDuration="1.472361563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:45.430015532 +0000 UTC m=+19.715550301" watchObservedRunningTime="2026-10-30 17:57:45.472361563 +0000 UTC m=+19.757896325" Oct 30 17:57:45.475262 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.475191 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ckrtw" Oct 30 17:57:45.478123 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.478098 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52cb20e_db43_407c_bf36_870475ceadb7.slice/crio-23fa282b57ed4364869fde9c5c71628742028abfd7982a22f947114a68929035 WatchSource:0}: Error finding container 23fa282b57ed4364869fde9c5c71628742028abfd7982a22f947114a68929035: Status 404 returned error can't find the container with id 23fa282b57ed4364869fde9c5c71628742028abfd7982a22f947114a68929035 Oct 30 17:57:45.483429 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.483406 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7d3975_f96e_4f38_9535_1f37d2142493.slice/crio-485291dd2b86da2b799fc215672372822537cb410dec2cafdad694a0a126966d WatchSource:0}: Error finding container 485291dd2b86da2b799fc215672372822537cb410dec2cafdad694a0a126966d: Status 404 returned error can't find the container with id 485291dd2b86da2b799fc215672372822537cb410dec2cafdad694a0a126966d Oct 30 17:57:45.483926 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.483904 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2daed88_3483_4e79_a6cc_f12cddc1bdac.slice/crio-44c98c0c8b57be286fbdd036ccf06de9be3c0cc1f4d25324d57f5606dfe5f0ec WatchSource:0}: Error finding container 44c98c0c8b57be286fbdd036ccf06de9be3c0cc1f4d25324d57f5606dfe5f0ec: Status 404 returned error can't find the container with id 44c98c0c8b57be286fbdd036ccf06de9be3c0cc1f4d25324d57f5606dfe5f0ec Oct 30 17:57:45.486721 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.486690 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7c72f6_d102_47f7_a69b_d720dac0fd6b.slice/crio-5f9aaf7d9414300042204ff8db23cf3e02438971ff6f984d37cc916e5b0147f4 WatchSource:0}: Error finding container 5f9aaf7d9414300042204ff8db23cf3e02438971ff6f984d37cc916e5b0147f4: Status 404 returned error can't find the container with id 5f9aaf7d9414300042204ff8db23cf3e02438971ff6f984d37cc916e5b0147f4 Oct 30 17:57:45.488633 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.488617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:57:45.498524 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:57:45.498502 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a9039c_06fd_4b76_adbd_92f5bb7e94b6.slice/crio-b291d1d26b946b36ac46dd56c03a8d44d2fc8ad74dc57cc9cfe4de699030e930 WatchSource:0}: Error finding container b291d1d26b946b36ac46dd56c03a8d44d2fc8ad74dc57cc9cfe4de699030e930: Status 404 returned error can't find the container with id b291d1d26b946b36ac46dd56c03a8d44d2fc8ad74dc57cc9cfe4de699030e930 Oct 30 17:57:45.789335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.789284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:45.789518 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.789357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:45.789518 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.789448 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.789518 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.789503 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:46.789485449 +0000 UTC m=+21.075020210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.789658 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.789448 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.789658 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.789601 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret podName:845bfa8a-e757-4cd4-bcd0-f3625bb21afa nodeName:}" failed. No retries permitted until 2026-10-30 17:57:46.78957357 +0000 UTC m=+21.075108321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret") pod "global-pull-secret-syncer-jg28z" (UID: "845bfa8a-e757-4cd4-bcd0-f3625bb21afa") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.834613 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.834567 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:45.891162 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:45.890481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:45.891162 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.890692 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:45.891162 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.890715 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:45.891162 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.890729 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nbjs2 for pod openshift-network-diagnostics/network-check-target-c5qcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:45.891162 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:45.890796 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2 podName:bd61c76f-9514-46db-99e2-21c6280a4842 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:46.890771536 +0000 UTC m=+21.176306297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjs2" (UniqueName: "kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2") pod "network-check-target-c5qcf" (UID: "bd61c76f-9514-46db-99e2-21c6280a4842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:46.149987 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.149898 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:45 +0000 UTC" deadline="2028-07-19 15:16:02.262563828 +0000 UTC" Oct 30 17:57:46.149987 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.149957 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15069h18m16.112610855s" Oct 30 17:57:46.224855 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.224449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:46.228363 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.224600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:46.260883 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.260773 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ckrtw" event={"ID":"fb7c72f6-d102-47f7-a69b-d720dac0fd6b","Type":"ContainerStarted","Data":"5f9aaf7d9414300042204ff8db23cf3e02438971ff6f984d37cc916e5b0147f4"} Oct 30 17:57:46.272629 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.272544 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" event={"ID":"b52cb20e-db43-407c-bf36-870475ceadb7","Type":"ContainerStarted","Data":"23fa282b57ed4364869fde9c5c71628742028abfd7982a22f947114a68929035"} Oct 30 17:57:46.298202 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.298167 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d65dr" event={"ID":"5dd334a8-7dd5-4a51-8ffc-adb0c815682f","Type":"ContainerStarted","Data":"b86c58491273d3225d53df5f0c2da6585fac542d2a64281a41773875a5984cfe"} Oct 30 17:57:46.304269 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.304205 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jtxd" event={"ID":"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a","Type":"ContainerStarted","Data":"347508d719c82b6df9a48faca4f859f6fe9dfa4279d07c6f6974c774567c7ea9"} Oct 30 17:57:46.314391 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.314114 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rfjbn" event={"ID":"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c","Type":"ContainerStarted","Data":"71ddb07e0b5f77310b67a3c582e4acff6545c9908588f9e1acffae511a57a597"} Oct 30 17:57:46.320039 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.319315 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal_9108d1d7bcbaabf4aba6b0607ef8c386/setup/0.log" Oct 30 17:57:46.320039 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.319349 2577 generic.go:356] "Generic (PLEG): container finished" podID="9108d1d7bcbaabf4aba6b0607ef8c386" containerID="2a00aa2391414eb1b75d1d9a6a793eae385d2d7f3a6912082486ed8c8783f2c7" exitCode=0 Oct 30 17:57:46.320039 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.319409 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerDied","Data":"2a00aa2391414eb1b75d1d9a6a793eae385d2d7f3a6912082486ed8c8783f2c7"} Oct 30 17:57:46.320039 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.319446 2577 scope.go:122] "RemoveContainer" containerID="f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261" Oct 30 17:57:46.320039 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.319809 2577 scope.go:122] "RemoveContainer" containerID="f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261" Oct 30 17:57:46.332279 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.332247 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"44c98c0c8b57be286fbdd036ccf06de9be3c0cc1f4d25324d57f5606dfe5f0ec"} Oct 30 17:57:46.341780 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.341757 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jrtkt" event={"ID":"6f7d3975-f96e-4f38-9535-1f37d2142493","Type":"ContainerStarted","Data":"485291dd2b86da2b799fc215672372822537cb410dec2cafdad694a0a126966d"} Oct 30 17:57:46.350301 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.350232 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5z8z" event={"ID":"e3952ef0-db1a-4c93-b9e4-fccda9618beb","Type":"ContainerStarted","Data":"1298430fd8f01d571a0d5ea01f613516e727d85850b746bc88a1349dc62c2e86"} Oct 30 17:57:46.362687 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.362538 2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal_openshift-machine-config-operator_9108d1d7bcbaabf4aba6b0607ef8c386_0 in pod sandbox e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16 from index: no such id: 'f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261'" containerID="f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261" Oct 30 17:57:46.362687 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.362590 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal_openshift-machine-config-operator_9108d1d7bcbaabf4aba6b0607ef8c386_0 in pod sandbox e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16 from index: no such id: 'f32b5864f056df890d6b287379abb3d261e1b892edcfc1068386b0935d8b6261'" Oct 30 17:57:46.370476 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.370400 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerStarted","Data":"b291d1d26b946b36ac46dd56c03a8d44d2fc8ad74dc57cc9cfe4de699030e930"} Oct 30 17:57:46.656277 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.656153 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:46.798139 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.798056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:46.798139 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.798119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:46.798361 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.798251 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:46.798361 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.798312 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:48.798294837 +0000 UTC m=+23.083829592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:46.798715 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.798668 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:46.798861 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.798722 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret podName:845bfa8a-e757-4cd4-bcd0-f3625bb21afa nodeName:}" failed. No retries permitted until 2026-10-30 17:57:48.798707505 +0000 UTC m=+23.084242256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret") pod "global-pull-secret-syncer-jg28z" (UID: "845bfa8a-e757-4cd4-bcd0-f3625bb21afa") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:46.899613 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:46.899099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:46.899613 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.899293 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:46.899613 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.899318 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:46.899613 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.899332 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nbjs2 for pod openshift-network-diagnostics/network-check-target-c5qcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:46.900870 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:46.900842 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2 podName:bd61c76f-9514-46db-99e2-21c6280a4842 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:48.899370869 +0000 UTC m=+23.184905637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjs2" (UniqueName: "kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2") pod "network-check-target-c5qcf" (UID: "bd61c76f-9514-46db-99e2-21c6280a4842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:47.150773 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:47.150709 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:45 +0000 UTC" deadline="2028-08-10 18:40:21.454334738 +0000 UTC" Oct 30 17:57:47.150773 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:47.150771 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15600h42m34.303567172s" Oct 30 17:57:47.219835 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:47.219786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:47.220005 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:47.219937 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:47.220287 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:47.220274 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:47.220386 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:47.220366 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:47.378080 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:47.378043 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerStarted","Data":"e1fb641f70ce6910f4630792708544220d56294f087b9c42176d34af3974ca9b"} Oct 30 17:57:47.421145 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:47.421044 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" podStartSLOduration=3.421026961 podStartE2EDuration="3.421026961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:47.420487705 +0000 UTC m=+21.706022480" watchObservedRunningTime="2026-10-30 17:57:47.421026961 +0000 UTC m=+21.706561731" Oct 30 17:57:48.222703 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:48.222672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:48.223249 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.222797 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:48.391192 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:48.390945 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="4e97187dc6fa69e4d11b9802aac18f3f3f0f39b8d9e6d6434b3b21e88ab8a56b" exitCode=0 Oct 30 17:57:48.391192 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:48.390998 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerDied","Data":"4e97187dc6fa69e4d11b9802aac18f3f3f0f39b8d9e6d6434b3b21e88ab8a56b"} Oct 30 17:57:48.813488 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:48.813403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:48.813488 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:48.813485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:48.813736 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.813665 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:48.813736 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.813726 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:52.813710494 +0000 UTC m=+27.099245244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:48.814189 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.814157 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:48.814270 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.814210 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret podName:845bfa8a-e757-4cd4-bcd0-f3625bb21afa nodeName:}" failed. No retries permitted until 2026-10-30 17:57:52.814194985 +0000 UTC m=+27.099729737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret") pod "global-pull-secret-syncer-jg28z" (UID: "845bfa8a-e757-4cd4-bcd0-f3625bb21afa") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:48.914768 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:48.914728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:48.914958 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.914934 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:48.914958 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.914955 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:48.915073 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.914967 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nbjs2 for pod openshift-network-diagnostics/network-check-target-c5qcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:48.915073 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:48.915026 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2 podName:bd61c76f-9514-46db-99e2-21c6280a4842 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:52.91500892 +0000 UTC m=+27.200543670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjs2" (UniqueName: "kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2") pod "network-check-target-c5qcf" (UID: "bd61c76f-9514-46db-99e2-21c6280a4842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:49.220543 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:49.220512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:49.220656 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:49.220639 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:49.221199 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:49.221010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:49.221199 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:49.221116 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:49.398338 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:49.398301 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jtxd" event={"ID":"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a","Type":"ContainerStarted","Data":"9d52fe120772bee7ff4850257ffaa2677a2f9a55a0dd539015b39793e846e89d"} Oct 30 17:57:50.224790 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:50.224721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:50.224945 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:50.224847 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:50.404935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:50.404896 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rfjbn" event={"ID":"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c","Type":"ContainerStarted","Data":"9d690bce576bcfc612dfd6eab5b828d551f5db82b0a3854bb64506872e46fc4e"} Oct 30 17:57:50.456320 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:50.456269 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:50.457480 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:50.457453 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:50.486613 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:50.486528 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5jtxd" podStartSLOduration=7.779790922 podStartE2EDuration="11.486508093s" totalImagesPullingTime="3.706717171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:50.446516813 +0000 UTC m=+24.732051583" watchObservedRunningTime="2026-10-30 17:57:50.486508093 +0000 UTC m=+24.772042864" Oct 30 17:57:50.529837 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:50.529771 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rfjbn" podStartSLOduration=7.072011087 podStartE2EDuration="11.529749366s" totalImagesPullingTime="4.457738279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:50.48728207 +0000 UTC m=+24.772816840" watchObservedRunningTime="2026-10-30 17:57:50.529749366 +0000 UTC m=+24.815284135" Oct 30 17:57:51.220322 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:51.220286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:51.220506 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:51.220419 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:51.220810 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:51.220791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:51.220937 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:51.220914 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:51.410577 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:51.410545 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" event={"ID":"b52cb20e-db43-407c-bf36-870475ceadb7","Type":"ContainerStarted","Data":"9a15e3731910ccd417f7923d165a8ae19e0a24a9ff8e5193104023c1e19d88fd"} Oct 30 17:57:52.220451 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:52.220409 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:52.220626 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.220549 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:52.413394 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:52.412834 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:57:52.480603 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:52.480345 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:57:52.850223 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:52.850138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:52.850223 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:52.850214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:52.850446 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.850365 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:52.850446 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.850437 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:00.850415665 +0000 UTC m=+35.135950429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:52.850561 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.850512 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:52.850561 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.850550 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret podName:845bfa8a-e757-4cd4-bcd0-f3625bb21afa nodeName:}" failed. No retries permitted until 2026-10-30 17:58:00.850538933 +0000 UTC m=+35.136073694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret") pod "global-pull-secret-syncer-jg28z" (UID: "845bfa8a-e757-4cd4-bcd0-f3625bb21afa") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:52.950996 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:52.950957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:52.951154 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.951105 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:52.951154 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.951127 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:52.951154 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.951139 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nbjs2 for pod openshift-network-diagnostics/network-check-target-c5qcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:52.951306 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:52.951197 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2 podName:bd61c76f-9514-46db-99e2-21c6280a4842 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:00.951178884 +0000 UTC m=+35.236713649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjs2" (UniqueName: "kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2") pod "network-check-target-c5qcf" (UID: "bd61c76f-9514-46db-99e2-21c6280a4842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:53.220520 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.220486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:53.220685 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.220531 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:53.220685 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:53.220598 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:53.220773 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:53.220737 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:53.290860 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.290732 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:57:52.48036225Z","UUID":"19eb3db8-f97f-4f87-84ae-3840fe611357","Handler":null,"Name":"","Endpoint":""} Oct 30 17:57:53.292296 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.292276 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:57:53.292397 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.292306 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:57:53.417545 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.417460 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="facd4d754f90dcdf7f1fc033b071ceab3a446a60b18f0d5d81f7be2dbc815b58" exitCode=0 Oct 30 17:57:53.418120 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.417553 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerDied","Data":"facd4d754f90dcdf7f1fc033b071ceab3a446a60b18f0d5d81f7be2dbc815b58"} Oct 30 17:57:53.420708 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.420691 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ckrtw" event={"ID":"fb7c72f6-d102-47f7-a69b-d720dac0fd6b","Type":"ContainerStarted","Data":"9aa11dedabd80ac8c50ab059d2467f9788af28e28997191982c0ad542add7726"} Oct 30 17:57:53.425482 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.425451 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" event={"ID":"b52cb20e-db43-407c-bf36-870475ceadb7","Type":"ContainerStarted","Data":"1f9a939c243dbe10a509de14c9a9861e92d8d24eaac6c64f787a6023737667a4"} Oct 30 17:57:53.506484 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:53.506300 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ckrtw" podStartSLOduration=6.995056762 podStartE2EDuration="14.506286341s" totalImagesPullingTime="7.511229579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:53.505711765 +0000 UTC m=+27.791246557" watchObservedRunningTime="2026-10-30 17:57:53.506286341 +0000 UTC m=+27.791821110" Oct 30 17:57:54.223172 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:54.223141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:54.223345 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:54.223268 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:54.430334 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:54.430297 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="b089a5808d4e410d87738f02e797914de63cadced1e408e44c26baabef2316e9" exitCode=0 Oct 30 17:57:54.430797 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:54.430391 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerDied","Data":"b089a5808d4e410d87738f02e797914de63cadced1e408e44c26baabef2316e9"} Oct 30 17:57:54.434846 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:54.434388 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" event={"ID":"b52cb20e-db43-407c-bf36-870475ceadb7","Type":"ContainerStarted","Data":"3859db990dcf103ce5dad19f04d8cf262abdc3bdf5eea63f9bcc13557c6b3222"} Oct 30 17:57:54.436320 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:54.436294 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jrtkt" event={"ID":"6f7d3975-f96e-4f38-9535-1f37d2142493","Type":"ContainerStarted","Data":"feaab5f3f8290a3a7f8fb189883010cbb3414f71e054e2cd7c20bd53ec1242cb"} Oct 30 17:57:54.521999 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:54.521952 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" podStartSLOduration=7.893945977 podStartE2EDuration="15.52193659s" totalImagesPullingTime="7.627990613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:54.521500101 +0000 UTC m=+28.807034872" watchObservedRunningTime="2026-10-30 17:57:54.52193659 +0000 UTC m=+28.807471388" Oct 30 17:57:55.219839 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.219801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:55.219959 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.219840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:55.220020 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:55.219964 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:55.220129 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:55.220108 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:55.439838 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.439783 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d65dr" event={"ID":"5dd334a8-7dd5-4a51-8ffc-adb0c815682f","Type":"ContainerStarted","Data":"506f21641aa1a8dbfd5fd4199bdad19ed7782f1e5c533921251bdadfdb9062b9"} Oct 30 17:57:55.441670 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.441633 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5z8z" event={"ID":"e3952ef0-db1a-4c93-b9e4-fccda9618beb","Type":"ContainerStarted","Data":"3b086324e11d00a9d227e1eb9788a606363067d35f3c94811c2e3c306bc9146c"} Oct 30 17:57:55.444485 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.444440 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="06f459c3ca91c117b589c216735ef06043de2a5ef90ae9f04556fe7c02a39888" exitCode=0 Oct 30 17:57:55.444615 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.444507 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerDied","Data":"06f459c3ca91c117b589c216735ef06043de2a5ef90ae9f04556fe7c02a39888"} Oct 30 17:57:55.483008 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.482966 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d65dr" podStartSLOduration=6.991407494 podStartE2EDuration="16.482954052s" totalImagesPullingTime="9.491546558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:55.482698931 +0000 UTC m=+29.768233702" watchObservedRunningTime="2026-10-30 17:57:55.482954052 +0000 UTC m=+29.768488820" Oct 30 17:57:55.483146 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.483086 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jrtkt" podStartSLOduration=8.631058932 podStartE2EDuration="16.483077052s" totalImagesPullingTime="7.85201812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:54.566922766 +0000 UTC m=+28.852457536" watchObservedRunningTime="2026-10-30 17:57:55.483077052 +0000 UTC m=+29.768611821" Oct 30 17:57:55.715879 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.715796 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:55.716014 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.715956 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:57:55.716494 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.716473 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:57:55.757935 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:55.757893 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c5z8z" podStartSLOduration=9.197215904 podStartE2EDuration="16.757880737s" totalImagesPullingTime="7.560664833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:55.569327434 +0000 UTC m=+29.854862206" watchObservedRunningTime="2026-10-30 17:57:55.757880737 +0000 UTC m=+30.043415505" Oct 30 17:57:56.220394 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:56.220363 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:56.220557 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:56.220457 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:57.220644 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:57.220553 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:57.220644 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:57.220603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:57.221150 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:57.220706 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:57.221150 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:57.220877 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:57.452231 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:57.452196 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="d5e39eaf50bb0e32298a0d6316ef611bf835501df6c956251c3d22facb444f87" exitCode=0 Oct 30 17:57:57.452391 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:57.452256 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerDied","Data":"d5e39eaf50bb0e32298a0d6316ef611bf835501df6c956251c3d22facb444f87"} Oct 30 17:57:58.220288 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:58.220258 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:57:58.220447 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:58.220394 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:57:58.457350 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:58.457317 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="a273ca0930ed8bb4c7f9056ee676ffe8d030033a5bbd04241bc02c3a69f0b849" exitCode=0 Oct 30 17:57:58.457887 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:58.457369 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerDied","Data":"a273ca0930ed8bb4c7f9056ee676ffe8d030033a5bbd04241bc02c3a69f0b849"} Oct 30 17:57:59.220042 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:59.220000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:57:59.220225 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:59.220000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:57:59.220225 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:59.220136 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:57:59.220225 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:57:59.220185 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:57:59.463075 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:59.463044 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86gbr" event={"ID":"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6","Type":"ContainerStarted","Data":"3f9a50f7c7485683efb27b1272020bc4ff77ec336572e31a420a6c6d9aca0535"} Oct 30 17:57:59.538367 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:57:59.538277 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-86gbr" podStartSLOduration=10.987950324 podStartE2EDuration="20.538263913s" totalImagesPullingTime="9.550313589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:59.524135033 +0000 UTC m=+33.809669803" watchObservedRunningTime="2026-10-30 17:57:59.538263913 +0000 UTC m=+33.823798679" Oct 30 17:58:00.222645 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.222446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:00.222739 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:00.222688 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:00.467930 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.467901 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"a3beac8d5351d3d0d40aff1f046953e721d432a9b16c5d3dae1ef4ae1034b4fd"} Oct 30 17:58:00.468519 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.467968 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"6d88a1dc2cd8a629a7a85557aa5c4489f8b2f906db625052e0dd7d51a3f7297b"} Oct 30 17:58:00.468519 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.468000 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"b3493150bd7021346569739b358b230bbaea5fc54832507cf767f72f832b4d4a"} Oct 30 17:58:00.468519 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.468028 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"0c37d4ac73b68feea367ee2d16016e04d0561e18f0fdaac5811a3789eda579ab"} Oct 30 17:58:00.468519 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.468053 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"9bbf648bd08c875d4f62989e355329eb3df8e6c5472ec1262c481b9168df952c"} Oct 30 17:58:00.468519 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.468079 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"0bb0fb3ae02a51f8f7e7de1da8425e16d7d9c0ffc338d618d894230653f73476"} Oct 30 17:58:00.914309 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.914279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:00.914469 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:00.914320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:00.914469 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:00.914406 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:58:00.914469 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:00.914406 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:58:00.914469 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:00.914451 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:16.914438756 +0000 UTC m=+51.199973516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:58:00.914469 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:00.914462 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret podName:845bfa8a-e757-4cd4-bcd0-f3625bb21afa nodeName:}" failed. No retries permitted until 2026-10-30 17:58:16.914457088 +0000 UTC m=+51.199991835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret") pod "global-pull-secret-syncer-jg28z" (UID: "845bfa8a-e757-4cd4-bcd0-f3625bb21afa") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:58:01.015027 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:01.014994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:01.015196 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:01.015137 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:58:01.015196 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:01.015157 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:58:01.015196 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:01.015168 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nbjs2 for pod openshift-network-diagnostics/network-check-target-c5qcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:58:01.015360 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:01.015223 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2 podName:bd61c76f-9514-46db-99e2-21c6280a4842 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:17.015207365 +0000 UTC m=+51.300742113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjs2" (UniqueName: "kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2") pod "network-check-target-c5qcf" (UID: "bd61c76f-9514-46db-99e2-21c6280a4842") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:58:01.219920 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:01.219843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:01.220065 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:01.219942 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:58:01.220065 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:01.220003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:01.220174 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:01.220109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:58:02.220475 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:02.220438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:02.221094 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:02.220560 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:02.475801 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:02.475730 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"dcf02faef0c1791646cd0a989c48196ad7212ab7afaa7b8b7c729f2df1ffb524"} Oct 30 17:58:03.220222 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:03.220190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:03.220416 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:03.220190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:03.220416 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:03.220285 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:58:03.220416 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:03.220391 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:58:04.222986 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:04.222953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:04.223440 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:04.223064 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:05.219955 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.219755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:05.220157 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.219755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:05.220157 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:05.220067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:58:05.220157 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:05.220091 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:58:05.484261 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.484191 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" event={"ID":"a2daed88-3483-4e79-a6cc-f12cddc1bdac","Type":"ContainerStarted","Data":"69464126e0d47a93a85d563211065b4d124243349ecfd8d2074eec7eb99e4c1d"} Oct 30 17:58:05.484681 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.484449 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:05.484681 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.484514 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:05.500643 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.500620 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:05.504220 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.504202 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:05.576203 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:05.576154 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" podStartSLOduration=12.095701992 podStartE2EDuration="26.576142746s" totalImagesPullingTime="14.480440754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:05.56195757 +0000 UTC m=+39.847492339" watchObservedRunningTime="2026-10-30 17:58:05.576142746 +0000 UTC m=+39.861677520" Oct 30 17:58:06.220999 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.220964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:06.221132 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:06.221071 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:06.487024 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.486958 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:06.625472 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.625440 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jg28z"] Oct 30 17:58:06.625622 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.625570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:06.625698 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:06.625673 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:58:06.629089 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.629063 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c5qcf"] Oct 30 17:58:06.629189 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.629145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:06.629240 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:06.629223 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:06.629543 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.629522 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prjhj"] Oct 30 17:58:06.629625 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:06.629615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:06.629702 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:06.629688 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:58:07.489232 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:07.489200 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:08.223125 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:08.223081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:08.223281 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:08.223163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:08.223281 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:08.223259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:08.223281 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:08.223260 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:58:08.223383 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:08.223331 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:58:08.223425 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:08.223405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:09.677346 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:09.677320 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:09.677740 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:09.677504 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:09.696835 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:09.696791 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" podUID="a2daed88-3483-4e79-a6cc-f12cddc1bdac" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:58:09.704846 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:09.704801 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" podUID="a2daed88-3483-4e79-a6cc-f12cddc1bdac" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:58:10.220375 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:10.220341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:10.220537 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:10.220390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:10.220537 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:10.220341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:10.220537 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:10.220459 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:58:10.220646 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:10.220548 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:58:10.220646 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:10.220590 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:10.821485 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:10.821316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:11.141024 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:11.140994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:11.324672 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:11.324638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:11.436633 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:11.436558 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:12.220443 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.220413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:12.220859 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.220466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:12.220859 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.220482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:12.220859 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:12.220555 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5qcf" podUID="bd61c76f-9514-46db-99e2-21c6280a4842" Oct 30 17:58:12.220859 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:12.220659 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jg28z" podUID="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" Oct 30 17:58:12.220859 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:12.220789 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prjhj" podUID="2cf30b64-3606-4e62-894c-1c5fbb2e7190" Oct 30 17:58:12.476408 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.476359 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeReady" Oct 30 17:58:12.476506 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.476476 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:58:12.546167 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.546140 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mmstz"] Oct 30 17:58:12.568033 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.568010 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ml8bz"] Oct 30 17:58:12.568142 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.568124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.584226 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.584198 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmstz"] Oct 30 17:58:12.584328 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.584248 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ml8bz"] Oct 30 17:58:12.584328 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.584314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:12.589322 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.589287 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:58:12.589322 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.589301 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:58:12.589603 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.589588 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdnw2\"" Oct 30 17:58:12.600750 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.600728 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:58:12.600885 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.600868 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:58:12.600942 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.600906 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:58:12.601059 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.601046 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8mdkh\"" Oct 30 17:58:12.701902 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.701874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0329ae-a289-4958-83b9-ff0c5455c068-config-volume\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.701902 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.701904 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f0329ae-a289-4958-83b9-ff0c5455c068-tmp-dir\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.702068 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.701920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b7e7-396a-454b-a7cc-94baedd50371-cert\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:12.702068 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.701935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.702068 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.702054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsz7\" (UniqueName: \"kubernetes.io/projected/eaf9b7e7-396a-454b-a7cc-94baedd50371-kube-api-access-2hsz7\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:12.702156 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.702086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flvvz\" (UniqueName: \"kubernetes.io/projected/3f0329ae-a289-4958-83b9-ff0c5455c068-kube-api-access-flvvz\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.770722 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.770661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:12.802745 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.802719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0329ae-a289-4958-83b9-ff0c5455c068-config-volume\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.802856 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.802750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f0329ae-a289-4958-83b9-ff0c5455c068-tmp-dir\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.802856 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.802766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b7e7-396a-454b-a7cc-94baedd50371-cert\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:12.802856 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.802782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.803001 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:12.802891 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:12.803001 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:12.802935 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls podName:3f0329ae-a289-4958-83b9-ff0c5455c068 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:13.302923899 +0000 UTC m=+47.588458650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls") pod "dns-default-mmstz" (UID: "3f0329ae-a289-4958-83b9-ff0c5455c068") : secret "dns-default-metrics-tls" not found Oct 30 17:58:12.803001 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.802940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsz7\" (UniqueName: \"kubernetes.io/projected/eaf9b7e7-396a-454b-a7cc-94baedd50371-kube-api-access-2hsz7\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:12.803151 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.803036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvvz\" (UniqueName: \"kubernetes.io/projected/3f0329ae-a289-4958-83b9-ff0c5455c068-kube-api-access-flvvz\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.803223 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.803206 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f0329ae-a289-4958-83b9-ff0c5455c068-tmp-dir\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.803328 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.803312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0329ae-a289-4958-83b9-ff0c5455c068-config-volume\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.806931 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.806907 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b7e7-396a-454b-a7cc-94baedd50371-cert\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:12.828257 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.828233 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsz7\" (UniqueName: \"kubernetes.io/projected/eaf9b7e7-396a-454b-a7cc-94baedd50371-kube-api-access-2hsz7\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:12.828335 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.828300 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvvz\" (UniqueName: \"kubernetes.io/projected/3f0329ae-a289-4958-83b9-ff0c5455c068-kube-api-access-flvvz\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:12.896601 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.896567 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:12.923711 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:12.923674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:13.278353 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:13.278318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:13.296434 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:13.296380 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ml8bz"] Oct 30 17:58:13.299322 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:58:13.299298 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf9b7e7_396a_454b_a7cc_94baedd50371.slice/crio-91d20aa529a1f4818f6aab1ad0912685b8780c97b8a918abdee9eb59948a7fce WatchSource:0}: Error finding container 91d20aa529a1f4818f6aab1ad0912685b8780c97b8a918abdee9eb59948a7fce: Status 404 returned error can't find the container with id 91d20aa529a1f4818f6aab1ad0912685b8780c97b8a918abdee9eb59948a7fce Oct 30 17:58:13.307402 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:13.307372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:13.307784 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:13.307760 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:13.307943 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:13.307874 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls podName:3f0329ae-a289-4958-83b9-ff0c5455c068 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.307847679 +0000 UTC m=+48.593382449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls") pod "dns-default-mmstz" (UID: "3f0329ae-a289-4958-83b9-ff0c5455c068") : secret "dns-default-metrics-tls" not found Oct 30 17:58:13.383408 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:13.383377 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:13.503212 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:13.503179 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ml8bz" event={"ID":"eaf9b7e7-396a-454b-a7cc-94baedd50371","Type":"ContainerStarted","Data":"91d20aa529a1f4818f6aab1ad0912685b8780c97b8a918abdee9eb59948a7fce"} Oct 30 17:58:14.219730 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.219696 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:14.219945 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.219708 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:14.219945 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.219863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:14.238721 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.238686 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:58:14.238721 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.238686 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:58:14.252293 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.252271 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2xlsx\"" Oct 30 17:58:14.252394 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.252349 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:58:14.252709 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.252694 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:58:14.252709 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.252701 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xl6hf\"" Oct 30 17:58:14.252805 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.252693 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6fpb9\"" Oct 30 17:58:14.312376 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.312342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:14.312836 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:14.312514 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:14.312836 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:14.312591 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls podName:3f0329ae-a289-4958-83b9-ff0c5455c068 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:16.312569332 +0000 UTC m=+50.598104098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls") pod "dns-default-mmstz" (UID: "3f0329ae-a289-4958-83b9-ff0c5455c068") : secret "dns-default-metrics-tls" not found Oct 30 17:58:14.469395 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.469340 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:14.566401 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:14.566327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:16.326015 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:16.325991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:16.326298 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:16.326125 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:16.326298 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:16.326176 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls podName:3f0329ae-a289-4958-83b9-ff0c5455c068 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:20.326164348 +0000 UTC m=+54.611699107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls") pod "dns-default-mmstz" (UID: "3f0329ae-a289-4958-83b9-ff0c5455c068") : secret "dns-default-metrics-tls" not found Oct 30 17:58:16.511611 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:16.511580 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ml8bz" event={"ID":"eaf9b7e7-396a-454b-a7cc-94baedd50371","Type":"ContainerStarted","Data":"62187d357593425bdda997efb300cee54f891c38f639bf776e4072f17d8a3893"} Oct 30 17:58:16.930202 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:16.930138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:16.930396 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:16.930223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:16.930396 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:16.930351 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:16.930515 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:16.930432 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:48.930413754 +0000 UTC m=+83.215948521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : secret "metrics-daemon-secret" not found Oct 30 17:58:16.932542 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:16.932516 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:16.943452 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:16.943432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:17.031458 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.031423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:17.033858 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.033837 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:17.036851 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.036803 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:17.179406 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.179375 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:17.183357 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.183285 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ml8bz" podStartSLOduration=2.418506304 podStartE2EDuration="5.183273154s" totalImagesPullingTime="2.76476685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:16.555137969 +0000 UTC m=+50.840672738" watchObservedRunningTime="2026-10-30 17:58:17.183273154 +0000 UTC m=+51.468807923" Oct 30 17:58:17.183671 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.183659 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg"] Oct 30 17:58:17.195778 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.195758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" Oct 30 17:58:17.207053 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.207031 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg"] Oct 30 17:58:17.220334 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.220315 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:17.220438 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.220420 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t4kbs\"" Oct 30 17:58:17.228218 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.228194 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:58:17.253144 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.253124 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jg28z"] Oct 30 17:58:17.256025 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:58:17.255999 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845bfa8a_e757_4cd4_bcd0_f3625bb21afa.slice/crio-282f89fc6274e9439fdc66059961674bd1fe310b79391587a40612bae9e2ffff WatchSource:0}: Error finding container 282f89fc6274e9439fdc66059961674bd1fe310b79391587a40612bae9e2ffff: Status 404 returned error can't find the container with id 282f89fc6274e9439fdc66059961674bd1fe310b79391587a40612bae9e2ffff Oct 30 17:58:17.281055 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.281036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:17.296662 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.296641 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:17.334254 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.334059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gxn\" (UniqueName: \"kubernetes.io/projected/5df9464e-c388-44ff-a808-e69b6dcf7d4d-kube-api-access-k7gxn\") pod \"migrator-66df5d496b-whsrg\" (UID: \"5df9464e-c388-44ff-a808-e69b6dcf7d4d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" Oct 30 17:58:17.350113 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.350088 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4d6r5"] Oct 30 17:58:17.368329 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.368308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.376169 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.376139 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4d6r5"] Oct 30 17:58:17.406798 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.406770 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:58:17.406798 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.406795 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mngtd\"" Oct 30 17:58:17.415013 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.414995 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:58:17.415013 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.414998 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:58:17.415467 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.415456 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:58:17.431580 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.431559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:17.431976 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.431961 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hn6rb-55v8r"] Oct 30 17:58:17.434648 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.434593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1489b258-153a-47ac-8740-c4e9417b90c6-crio-socket\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.434648 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.434646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gxn\" (UniqueName: \"kubernetes.io/projected/5df9464e-c388-44ff-a808-e69b6dcf7d4d-kube-api-access-k7gxn\") pod \"migrator-66df5d496b-whsrg\" (UID: \"5df9464e-c388-44ff-a808-e69b6dcf7d4d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" Oct 30 17:58:17.434791 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.434677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.434791 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.434777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbbb\" (UniqueName: \"kubernetes.io/projected/1489b258-153a-47ac-8740-c4e9417b90c6-kube-api-access-scbbb\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.434885 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.434802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1489b258-153a-47ac-8740-c4e9417b90c6-data-volume\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.434885 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.434846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.444598 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.444575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.455701 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.455683 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hn6rb-55v8r"] Oct 30 17:58:17.459638 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.459620 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gxn\" (UniqueName: \"kubernetes.io/projected/5df9464e-c388-44ff-a808-e69b6dcf7d4d-kube-api-access-k7gxn\") pod \"migrator-66df5d496b-whsrg\" (UID: \"5df9464e-c388-44ff-a808-e69b6dcf7d4d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" Oct 30 17:58:17.461635 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.461619 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-swnsr\"" Oct 30 17:58:17.515310 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.515291 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:58:17.515447 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.515427 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jg28z" event={"ID":"845bfa8a-e757-4cd4-bcd0-f3625bb21afa","Type":"ContainerStarted","Data":"282f89fc6274e9439fdc66059961674bd1fe310b79391587a40612bae9e2ffff"} Oct 30 17:58:17.519954 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.519939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" Oct 30 17:58:17.530235 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.530208 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:58:17.531184 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.531163 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:17.536083 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.536205 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.536205 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.536205 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:17.536150 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:17.536464 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:17.536250 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.036230393 +0000 UTC m=+52.321765164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : configmap "kube-rbac-proxy" not found Oct 30 17:58:17.536464 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1489b258-153a-47ac-8740-c4e9417b90c6-crio-socket\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.536464 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljf9\" (UniqueName: \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.536464 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.536653 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbbb\" (UniqueName: \"kubernetes.io/projected/1489b258-153a-47ac-8740-c4e9417b90c6-kube-api-access-scbbb\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.536653 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:17.536576 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:17.536653 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1489b258-153a-47ac-8740-c4e9417b90c6-crio-socket\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.536653 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:17.536621 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.036607079 +0000 UTC m=+52.322141841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:17.536653 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.536897 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1489b258-153a-47ac-8740-c4e9417b90c6-data-volume\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.537014 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.536999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1489b258-153a-47ac-8740-c4e9417b90c6-data-volume\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.557640 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.557618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbbb\" (UniqueName: \"kubernetes.io/projected/1489b258-153a-47ac-8740-c4e9417b90c6-kube-api-access-scbbb\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:17.589144 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:58:17.589119 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd61c76f_9514_46db_99e2_21c6280a4842.slice/crio-4626f52e99f06ab0530b7b5c7ee4bf18ccc399305a7adbdb15da4a59c0117687 WatchSource:0}: Error finding container 4626f52e99f06ab0530b7b5c7ee4bf18ccc399305a7adbdb15da4a59c0117687: Status 404 returned error can't find the container with id 4626f52e99f06ab0530b7b5c7ee4bf18ccc399305a7adbdb15da4a59c0117687 Oct 30 17:58:17.593844 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.593807 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c5qcf"] Oct 30 17:58:17.637184 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.637146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.637184 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.637187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.637535 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.637514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljf9\" (UniqueName: \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.637648 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.637632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.637892 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.637856 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.638004 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.637974 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.638211 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.638193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.659720 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.659694 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljf9\" (UniqueName: \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.720237 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.720167 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-csb7v"] Oct 30 17:58:17.733474 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.733446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.745365 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.745341 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-csb7v"] Oct 30 17:58:17.750001 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.749979 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:58:17.750140 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.750127 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:58:17.750219 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.750158 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:58:17.750219 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.750194 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7dzl7\"" Oct 30 17:58:17.750427 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.750413 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:58:17.766176 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.766153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:17.825358 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.825327 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg"] Oct 30 17:58:17.828204 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:58:17.828174 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df9464e_c388_44ff_a808_e69b6dcf7d4d.slice/crio-5451f7b3cbd6c6946950be2175fbf01234994fd4dd4891f9404d19a83fd4a70b WatchSource:0}: Error finding container 5451f7b3cbd6c6946950be2175fbf01234994fd4dd4891f9404d19a83fd4a70b: Status 404 returned error can't find the container with id 5451f7b3cbd6c6946950be2175fbf01234994fd4dd4891f9404d19a83fd4a70b Oct 30 17:58:17.839979 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.839947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-cabundle\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.840088 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.840000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-key\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.840088 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.840073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzh9v\" (UniqueName: \"kubernetes.io/projected/a0113161-09c9-4be8-9e2d-d9dd939b7593-kube-api-access-tzh9v\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.940926 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.940892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-cabundle\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.941080 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.940940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-key\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.941080 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.940979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzh9v\" (UniqueName: \"kubernetes.io/projected/a0113161-09c9-4be8-9e2d-d9dd939b7593-kube-api-access-tzh9v\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.941609 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.941582 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-cabundle\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.943629 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.943608 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-key\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:17.962754 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:17.962723 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzh9v\" (UniqueName: \"kubernetes.io/projected/a0113161-09c9-4be8-9e2d-d9dd939b7593-kube-api-access-tzh9v\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:18.041691 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.041621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:18.041808 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.041701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:18.041808 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:18.041803 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:18.041901 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:18.041872 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:19.041855374 +0000 UTC m=+53.327390138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:18.041901 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:18.041891 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:18.041965 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:18.041932 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:19.041919311 +0000 UTC m=+53.327454074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : configmap "kube-rbac-proxy" not found Oct 30 17:58:18.058367 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.058336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:18.084383 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.084296 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hn6rb-55v8r"] Oct 30 17:58:18.089525 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:58:18.089493 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02d9ec7_7d1e_4d1a_a24e_1bf17f279022.slice/crio-347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f WatchSource:0}: Error finding container 347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f: Status 404 returned error can't find the container with id 347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f Oct 30 17:58:18.152450 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.152389 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:18.253294 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.253214 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:18.375667 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.375637 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-csb7v"] Oct 30 17:58:18.380690 ip-10-0-133-124 kubenswrapper[2577]: W1030 17:58:18.380650 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0113161_09c9_4be8_9e2d_d9dd939b7593.slice/crio-8eabc5fe986e84f49724075384d88321882b639a30b3d0a566571d5532e5de47 WatchSource:0}: Error finding container 8eabc5fe986e84f49724075384d88321882b639a30b3d0a566571d5532e5de47: Status 404 returned error can't find the container with id 8eabc5fe986e84f49724075384d88321882b639a30b3d0a566571d5532e5de47 Oct 30 17:58:18.519734 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.519704 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" event={"ID":"a0113161-09c9-4be8-9e2d-d9dd939b7593","Type":"ContainerStarted","Data":"8eabc5fe986e84f49724075384d88321882b639a30b3d0a566571d5532e5de47"} Oct 30 17:58:18.521512 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.521482 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" event={"ID":"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022","Type":"ContainerStarted","Data":"347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f"} Oct 30 17:58:18.522848 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.522806 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" event={"ID":"5df9464e-c388-44ff-a808-e69b6dcf7d4d","Type":"ContainerStarted","Data":"5451f7b3cbd6c6946950be2175fbf01234994fd4dd4891f9404d19a83fd4a70b"} Oct 30 17:58:18.524562 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.524504 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c5qcf" event={"ID":"bd61c76f-9514-46db-99e2-21c6280a4842","Type":"ContainerStarted","Data":"4626f52e99f06ab0530b7b5c7ee4bf18ccc399305a7adbdb15da4a59c0117687"} Oct 30 17:58:18.736040 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:18.735928 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:19.050431 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:19.050343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:19.050431 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:19.050417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:19.050609 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:19.050559 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:19.050643 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:19.050620 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:21.05060154 +0000 UTC m=+55.336136294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : configmap "kube-rbac-proxy" not found Oct 30 17:58:19.051057 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:19.051035 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:19.051125 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:19.051090 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:21.051078845 +0000 UTC m=+55.336613605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:20.361384 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:20.361347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:20.362031 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:20.361505 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:20.362031 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:20.361571 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls podName:3f0329ae-a289-4958-83b9-ff0c5455c068 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:28.361552618 +0000 UTC m=+62.647087378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls") pod "dns-default-mmstz" (UID: "3f0329ae-a289-4958-83b9-ff0c5455c068") : secret "dns-default-metrics-tls" not found Oct 30 17:58:20.532468 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:20.532425 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" event={"ID":"5df9464e-c388-44ff-a808-e69b6dcf7d4d","Type":"ContainerStarted","Data":"595b3f7d3725f15aafb8232a6e46759767b6fd0134942b65aafce4476e76e004"} Oct 30 17:58:21.067697 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:21.067655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:21.067907 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:21.067772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:21.067986 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:21.067939 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:21.068041 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:21.068006 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.067987969 +0000 UTC m=+59.353522768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:21.068101 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:21.068062 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:21.068101 ip-10-0-133-124 kubenswrapper[2577]: E1030 17:58:21.068093 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.068085899 +0000 UTC m=+59.353620645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : configmap "kube-rbac-proxy" not found Oct 30 17:58:21.537481 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:21.537428 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c5qcf" event={"ID":"bd61c76f-9514-46db-99e2-21c6280a4842","Type":"ContainerStarted","Data":"135c726066cd62ba507302fab8cee0367b5c944c2237f8a4d9a6e32f8bda4bec"} Oct 30 17:58:21.539889 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:21.539524 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" event={"ID":"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022","Type":"ContainerStarted","Data":"b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698"} Oct 30 17:58:21.584102 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:21.582888 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" podStartSLOduration=2.667323854 podStartE2EDuration="4.582872284s" totalImagesPullingTime="1.91554843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:20.585216163 +0000 UTC m=+54.870750935" watchObservedRunningTime="2026-10-30 17:58:21.582872284 +0000 UTC m=+55.868407055" Oct 30 17:58:21.584102 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:21.583390 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" podStartSLOduration=1.519883855 podStartE2EDuration="4.583377825s" totalImagesPullingTime="3.06349397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:21.582620185 +0000 UTC m=+55.868154958" watchObservedRunningTime="2026-10-30 17:58:21.583377825 +0000 UTC m=+55.868912608" Oct 30 17:58:21.958798 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:21.958758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:22.327143 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:22.327062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:22.530401 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:22.530363 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:22.548839 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:22.547628 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" event={"ID":"a0113161-09c9-4be8-9e2d-d9dd939b7593","Type":"ContainerStarted","Data":"53f50c8fee43e4ef0bddff6fdd1c7e5512a279b9af67cf95e11ecda4c54e2b25"} Oct 30 17:58:22.649393 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:22.649346 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c5qcf" podStartSLOduration=39.770829795 podStartE2EDuration="43.649332098s" totalImagesPullingTime="3.878502303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:22.648340397 +0000 UTC m=+56.933875165" watchObservedRunningTime="2026-10-30 17:58:22.649332098 +0000 UTC m=+56.934866866" Oct 30 17:58:22.649575 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:22.649559 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" podStartSLOduration=2.496249898 podStartE2EDuration="5.649552527s" totalImagesPullingTime="3.153302629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:22.599447988 +0000 UTC m=+56.884982738" watchObservedRunningTime="2026-10-30 17:58:22.649552527 +0000 UTC m=+56.935087304" Oct 30 17:58:22.714333 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:22.714296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:22.836747 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:22.836716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:23.023647 ip-10-0-133-124 kubenswrapper[2577]: I1030 17:58:23.023597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:23.101199 ip-10-0-133-124 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:58:23.115283 ip-10-0-133-124 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:58:23.115452 ip-10-0-133-124 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:58:23.115623 ip-10-0-133-124 systemd[1]: kubelet.service: Consumed 2.677s CPU time. Oct 30 17:58:23.132430 ip-10-0-133-124 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:58:23.205565 ip-10-0-133-124 kubenswrapper[4582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:23.205565 ip-10-0-133-124 kubenswrapper[4582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:58:23.205565 ip-10-0-133-124 kubenswrapper[4582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:23.205565 ip-10-0-133-124 kubenswrapper[4582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208155 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208259 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208272 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208277 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208282 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208287 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208292 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208297 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208301 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208305 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208309 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208314 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208319 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208323 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208327 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208337 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208341 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208347 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208351 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.208495 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208356 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208362 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208365 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208370 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208376 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208380 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208384 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208388 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208393 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208402 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208406 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208410 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208415 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208420 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208424 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208428 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208432 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208437 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208441 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208445 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.209094 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208449 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208458 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208462 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208466 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208471 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208475 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208480 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208503 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208525 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208531 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208537 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208541 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208546 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208551 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208556 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208560 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208565 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208569 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208573 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.209707 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208586 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208602 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208608 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208613 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208618 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208622 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208626 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208630 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208636 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208640 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208645 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208649 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208675 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208683 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208686 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208689 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208692 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208695 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208698 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.210264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208702 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208707 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208710 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208713 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208716 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208719 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208725 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208730 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208734 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208737 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208740 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208744 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208747 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208750 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208753 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208755 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208758 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208764 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208767 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208770 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.210729 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208773 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208775 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208778 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208781 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208784 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208791 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208795 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208800 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.208804 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210239 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210251 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210255 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210258 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210261 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210265 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210268 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210270 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210273 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210276 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.211284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210279 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210282 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210285 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210287 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210290 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210293 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210296 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210298 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210301 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210304 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210307 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210310 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210313 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210316 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210319 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210322 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210325 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210327 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210330 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210333 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.211747 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210336 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210339 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210342 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210344 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210347 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210349 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210352 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210355 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210357 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210360 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210363 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210365 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210368 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210371 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210374 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210377 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210379 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210382 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210384 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210387 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.212288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210390 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210392 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210395 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210397 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210400 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210403 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210406 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210408 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210411 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210413 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210416 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210419 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210421 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210424 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210427 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210430 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210432 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210435 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210438 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210440 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.212768 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210443 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210446 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210449 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210452 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210454 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210457 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210461 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210464 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210467 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210470 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210473 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210476 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210478 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210481 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210484 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210486 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210489 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210492 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.213288 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210496 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210501 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210504 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210507 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210510 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210512 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210515 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210518 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210520 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210523 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210536 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210541 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210545 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210548 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210551 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210554 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210557 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.210560 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211050 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.213728 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211056 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211059 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211062 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211065 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211068 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211070 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211073 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211076 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211079 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211081 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211084 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211088 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211091 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211094 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211120 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211124 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211127 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211130 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211133 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211136 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.214202 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211139 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211142 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211144 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211147 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211150 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211153 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211156 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211158 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211161 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211163 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211166 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211169 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211172 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211175 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211178 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211181 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211184 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211186 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211189 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.214685 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211192 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211195 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211198 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211201 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211203 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211206 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211209 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211211 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211214 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211216 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211219 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211222 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211225 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211227 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211232 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211236 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211239 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211242 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211245 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.215432 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211247 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211250 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211253 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211256 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211259 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211261 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211264 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211266 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211269 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211273 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211276 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211278 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211281 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211284 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211288 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211292 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211295 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211298 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211301 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.216271 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211303 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211306 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211308 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211311 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211314 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211317 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211319 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211322 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211325 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211328 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211330 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211333 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211336 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211339 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211342 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211345 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211348 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211350 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211366 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.217098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211369 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211372 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211375 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211378 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211381 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211384 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211387 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211390 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.211392 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211479 4582 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211486 4582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211492 4582 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211498 4582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211502 4582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211505 4582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211509 4582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211514 4582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211517 4582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211520 4582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211524 4582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211527 4582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211530 4582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:58:23.218026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211533 4582 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211536 4582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211541 4582 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211544 4582 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211547 4582 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211551 4582 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211554 4582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211558 4582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211561 4582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211564 4582 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211568 4582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211574 4582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211577 4582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211581 4582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211584 4582 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211587 4582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211590 4582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211594 4582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211598 4582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211603 4582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211606 4582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211633 4582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211637 4582 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211641 4582 flags.go:64] FLAG: --enable-server="true" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211644 4582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:58:23.219022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211649 4582 flags.go:64] FLAG: --event-burst="100" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211652 4582 flags.go:64] FLAG: --event-qps="50" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211655 4582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211658 4582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211661 4582 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211666 4582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211669 4582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211672 4582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211675 4582 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211678 4582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211681 4582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211685 4582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211687 4582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211690 4582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211693 4582 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211697 4582 flags.go:64] FLAG: --feature-gates="" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211701 4582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211705 4582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211709 4582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211713 4582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211716 4582 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211719 4582 flags.go:64] FLAG: --help="false" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211722 4582 flags.go:64] FLAG: --hostname-override="ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211726 4582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:58:23.220174 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211729 4582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211734 4582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211737 4582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211741 4582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211744 4582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211747 4582 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211750 4582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211753 4582 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211756 4582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211759 4582 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211762 4582 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211765 4582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211768 4582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211772 4582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211775 4582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211778 4582 flags.go:64] FLAG: --lock-file="" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211781 4582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211784 4582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211787 4582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211792 4582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211795 4582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211799 4582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211802 4582 flags.go:64] FLAG: --logging-format="text" Oct 30 17:58:23.221202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211805 4582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211808 4582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211811 4582 flags.go:64] FLAG: --manifest-url="" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211815 4582 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211835 4582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211839 4582 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211843 4582 flags.go:64] FLAG: --max-pods="110" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211846 4582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211849 4582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211853 4582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211858 4582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211861 4582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211864 4582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211867 4582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211896 4582 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211899 4582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211903 4582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211906 4582 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211910 4582 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211913 4582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211916 4582 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211919 4582 flags.go:64] FLAG: --port="10250" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211923 4582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211926 4582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0432ec29561a4d6c8" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211929 4582 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211932 4582 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:58:23.222209 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211935 4582 flags.go:64] FLAG: --register-node="true" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211938 4582 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211946 4582 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211949 4582 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211952 4582 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211955 4582 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211959 4582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211962 4582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211965 4582 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211970 4582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211973 4582 flags.go:64] FLAG: --runonce="false" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211976 4582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211979 4582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211983 4582 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211986 4582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211989 4582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.211996 4582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212000 4582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212003 4582 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212006 4582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212010 4582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212013 4582 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212016 4582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212019 4582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212022 4582 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:58:23.223401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212025 4582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212030 4582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212033 4582 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212036 4582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212041 4582 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212044 4582 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212047 4582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212050 4582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212053 4582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212056 4582 flags.go:64] FLAG: --v="2" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212061 4582 flags.go:64] FLAG: --version="false" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212065 4582 flags.go:64] FLAG: --vmodule="" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212069 4582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212072 4582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212204 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212208 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212213 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212216 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212219 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212224 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212228 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212231 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212234 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.224303 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212238 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212242 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212245 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212247 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212250 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212253 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212256 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212258 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212261 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212264 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212267 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212269 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212272 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212275 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212277 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212280 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212283 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212285 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212289 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212292 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.224976 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212295 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212297 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212300 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212303 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212306 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212310 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212312 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212315 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212318 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212320 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212323 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212326 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212330 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212332 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212336 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212338 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212341 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212344 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212347 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.225616 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212350 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212352 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212355 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212358 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212360 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212363 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212366 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212368 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212371 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212374 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212376 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212381 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212384 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212388 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212391 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212394 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212396 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212399 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212403 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.226292 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212406 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212409 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212411 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212414 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212417 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212420 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212424 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212426 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212429 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212433 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212435 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212438 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212441 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212443 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212446 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212449 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212452 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212454 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212457 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.227171 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212460 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212462 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212465 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212468 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212471 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212474 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212476 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212479 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212482 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212484 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212487 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212490 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212494 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212497 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212499 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212502 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212505 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212507 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212510 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.227811 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212514 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.212541 4582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212580 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212584 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212588 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212591 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212594 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212597 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212599 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212602 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212605 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212607 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212610 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212613 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212617 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212621 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.228765 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212624 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212627 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212631 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212633 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212636 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212639 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212642 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212645 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212648 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212652 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212655 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212657 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212660 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212663 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212666 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212669 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212671 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212675 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212678 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.229481 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212680 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212683 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212686 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212689 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212692 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212694 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212697 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212700 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212703 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212705 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212708 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212711 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212714 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212716 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212719 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212721 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212724 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212727 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212729 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212732 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.230309 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212735 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212738 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212742 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212744 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212747 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212749 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212752 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212755 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212757 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212760 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212764 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212769 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212773 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212776 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212779 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212782 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212785 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212788 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212790 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212793 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.231223 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212796 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212798 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212801 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212804 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212807 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212810 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212812 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212815 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212832 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212837 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212841 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212843 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212846 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212849 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212853 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212856 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212859 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212861 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212864 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212867 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.232098 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212869 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212872 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212877 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212879 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212882 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212885 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212888 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212890 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212894 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212896 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212899 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212902 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.212905 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.217323 4582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.217343 4582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.217369 4582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.217376 4582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217458 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217466 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.232966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217471 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217476 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217481 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217486 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217490 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217494 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217499 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217503 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217508 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217512 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217516 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217521 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217526 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217530 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217535 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217539 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217543 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217548 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217552 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.233784 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217557 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217561 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217566 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217570 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217575 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217579 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217583 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217588 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217593 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217598 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217602 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217607 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217611 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217615 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217619 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217623 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217628 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217632 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217637 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217641 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.234710 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217646 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217650 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217654 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217658 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217663 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217668 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217672 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217676 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217681 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217686 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217690 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217695 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217700 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217705 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217709 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217713 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217717 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217721 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217726 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217731 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.235567 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217735 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217741 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217746 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217750 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217754 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217759 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217763 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217767 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217772 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217776 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217780 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217785 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217789 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217793 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217798 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217802 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217807 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217811 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217832 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.236555 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217837 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217841 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217845 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217849 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217853 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217857 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217860 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217864 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217868 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217872 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217876 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217882 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217887 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217892 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217897 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217902 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217906 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217911 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217915 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217920 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.237391 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217924 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217931 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217937 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217941 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217946 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.217953 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.217994 4582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218214 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218223 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218228 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218233 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218238 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218242 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218247 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218251 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.238264 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218255 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218260 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218264 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218268 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218273 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218277 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218281 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218285 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218290 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218295 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218299 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218306 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218312 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218318 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218323 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218328 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218332 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218337 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218341 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.238940 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218345 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218349 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218354 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218359 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218363 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218367 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218371 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218375 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218380 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218384 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218388 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218393 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218398 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218402 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218408 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218414 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218420 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218424 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218429 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.239749 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218433 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218439 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218444 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218449 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218453 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218457 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218462 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218466 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218496 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218504 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218509 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218514 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218518 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218522 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218527 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218531 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218535 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218539 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218544 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218548 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.240586 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218552 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218557 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218561 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218565 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218570 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218575 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218579 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218583 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218588 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218592 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218596 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218601 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218605 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218609 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218614 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218619 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218623 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218628 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218632 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.241494 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218636 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218641 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218646 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218650 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218654 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218659 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218664 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218669 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218673 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218678 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218682 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218686 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218691 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218695 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218699 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218704 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218709 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218713 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218717 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218721 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.242030 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218726 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.218763 4582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218839 4582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218848 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218853 4582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218858 4582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218862 4582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218867 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218872 4582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218876 4582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218883 4582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218890 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218895 4582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218902 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218907 4582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:23.242556 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218913 4582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218918 4582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218926 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218930 4582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218935 4582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218939 4582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218944 4582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218949 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218953 4582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218957 4582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218962 4582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218966 4582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218970 4582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218975 4582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218979 4582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218983 4582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218987 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.218994 4582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219000 4582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:23.242953 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219005 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219010 4582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219017 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219022 4582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219026 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219030 4582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219034 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219039 4582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219043 4582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219047 4582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219051 4582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219056 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219061 4582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219065 4582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219070 4582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219077 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219081 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219085 4582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219090 4582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219094 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:23.243417 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219099 4582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219103 4582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219108 4582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219112 4582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219117 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219121 4582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219126 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219130 4582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219134 4582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219138 4582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219143 4582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219147 4582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219152 4582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219156 4582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219162 4582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219166 4582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219172 4582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219176 4582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219180 4582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219185 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:23.243966 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219189 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219194 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219198 4582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219203 4582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219207 4582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219211 4582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219216 4582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219222 4582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219228 4582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219232 4582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219237 4582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219241 4582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219245 4582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219250 4582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219254 4582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219259 4582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219263 4582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219267 4582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219271 4582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219276 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:23.244426 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219280 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219285 4582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219289 4582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219294 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219298 4582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219303 4582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219309 4582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219313 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219318 4582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219322 4582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219327 4582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219331 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219337 4582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:23.219342 4582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.219431 4582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.220948 4582 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.221022 4582 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.221295 4582 server.go:1014] "Starting client certificate rotation" Oct 30 17:58:23.244899 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.221421 4582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.221462 4582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:52:36 +0000 UTC" deadline="2028-05-20 00:48:41.762404577 +0000 UTC" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.221497 4582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13614h50m18.540909603s" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.222011 4582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.223612 4582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.225331 4582 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.227314 4582 log.go:25] "Validated CRI v1 image API" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.228842 4582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:58:23.245411 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.231483 4582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 873a89f6-9599-4b6d-96a7-48069d8ff286:/dev/nvme0n1p3 957d7702-5b85-4489-86b0-843ba1e18de3:/dev/nvme0n1p4] Oct 30 17:58:23.245627 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.231499 4582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1298430fd8f01d571a0d5ea01f613516e727d85850b746bc88a1349dc62c2e86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1298430fd8f01d571a0d5ea01f613516e727d85850b746bc88a1349dc62c2e86/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23fa282b57ed4364869fde9c5c71628742028abfd7982a22f947114a68929035/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23fa282b57ed4364869fde9c5c71628742028abfd7982a22f947114a68929035/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/282f89fc6274e9439fdc66059961674bd1fe310b79391587a40612bae9e2ffff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/282f89fc6274e9439fdc66059961674bd1fe310b79391587a40612bae9e2ffff/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/347508d719c82b6df9a48faca4f859f6fe9dfa4279d07c6f6974c774567c7ea9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/347508d719c82b6df9a48faca4f859f6fe9dfa4279d07c6f6974c774567c7ea9/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44c98c0c8b57be286fbdd036ccf06de9be3c0cc1f4d25324d57f5606dfe5f0ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44c98c0c8b57be286fbdd036ccf06de9be3c0cc1f4d25324d57f5606dfe5f0ec/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4626f52e99f06ab0530b7b5c7ee4bf18ccc399305a7adbdb15da4a59c0117687/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4626f52e99f06ab0530b7b5c7ee4bf18ccc399305a7adbdb15da4a59c0117687/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/485291dd2b86da2b799fc215672372822537cb410dec2cafdad694a0a126966d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/485291dd2b86da2b799fc215672372822537cb410dec2cafdad694a0a126966d/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5451f7b3cbd6c6946950be2175fbf01234994fd4dd4891f9404d19a83fd4a70b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5451f7b3cbd6c6946950be2175fbf01234994fd4dd4891f9404d19a83fd4a70b/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f9aaf7d9414300042204ff8db23cf3e02438971ff6f984d37cc916e5b0147f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f9aaf7d9414300042204ff8db23cf3e02438971ff6f984d37cc916e5b0147f4/userdata/shm major:0 minor:99 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71ddb07e0b5f77310b67a3c582e4acff6545c9908588f9e1acffae511a57a597/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71ddb07e0b5f77310b67a3c582e4acff6545c9908588f9e1acffae511a57a597/userdata/shm major:0 minor:55 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8eabc5fe986e84f49724075384d88321882b639a30b3d0a566571d5532e5de47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8eabc5fe986e84f49724075384d88321882b639a30b3d0a566571d5532e5de47/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91d20aa529a1f4818f6aab1ad0912685b8780c97b8a918abdee9eb59948a7fce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91d20aa529a1f4818f6aab1ad0912685b8780c97b8a918abdee9eb59948a7fce/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b291d1d26b946b36ac46dd56c03a8d44d2fc8ad74dc57cc9cfe4de699030e930/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b291d1d26b946b36ac46dd56c03a8d44d2fc8ad74dc57cc9cfe4de699030e930/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1489b258-153a-47ac-8740-c4e9417b90c6/volumes/kubernetes.io~projected/kube-api-access-scbbb:{mountpoint:/var/lib/kubelet/pods/1489b258-153a-47ac-8740-c4e9417b90c6/volumes/kubernetes.io~projected/kube-api-access-scbbb major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cf30b64-3606-4e62-894c-1c5fbb2e7190/volumes/kubernetes.io~projected/kube-api-access-g6hbk:{mountpoint:/var/lib/kubelet/pods/2cf30b64-3606-4e62-894c-1c5fbb2e7190/volumes/kubernetes.io~projected/kube-api-access-g6hbk major:0 minor:73 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f0329ae-a289-4958-83b9-ff0c5455c068/volumes/kubernetes.io~projected/kube-api-access-flvvz:{mountpoint:/var/lib/kubelet/pods/3f0329ae-a289-4958-83b9-ff0c5455c068/volumes/kubernetes.io~projected/kube-api-access-flvvz major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~projected/kube-api-access-ncnxb:{mountpoint:/var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~projected/kube-api-access-ncnxb major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5df9464e-c388-44ff-a808-e69b6dcf7d4d/volumes/kubernetes.io~projected/kube-api-access-k7gxn:{mountpoint:/var/lib/kubelet/pods/5df9464e-c388-44ff-a808-e69b6dcf7d4d/volumes/kubernetes.io~projected/kube-api-access-k7gxn major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f7d3975-f96e-4f38-9535-1f37d2142493/volumes/kubernetes.io~projected/kube-api-access-h9l7f:{mountpoint:/var/lib/kubelet/pods/6f7d3975-f96e-4f38-9535-1f37d2142493/volumes/kubernetes.io~projected/kube-api-access-h9l7f major:0 minor:76 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/845bfa8a-e757-4cd4-bcd0-f3625bb21afa/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/845bfa8a-e757-4cd4-bcd0-f3625bb21afa/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0113161-09c9-4be8-9e2d-d9dd939b7593/volumes/kubernetes.io~projected/kube-api-access-tzh9v:{mountpoint:/var/lib/kubelet/pods/a0113161-09c9-4be8-9e2d-d9dd939b7593/volumes/kubernetes.io~projected/kube-api-access-tzh9v major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0113161-09c9-4be8-9e2d-d9dd939b7593/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/a0113161-09c9-4be8-9e2d-d9dd939b7593/volumes/kubernetes.io~secret/signing-key major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022/volumes/kubernetes.io~projected/kube-api-access-4ljf9:{mountpoint:/var/lib/kubelet/pods/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022/volumes/kubernetes.io~projected/kube-api-access-4ljf9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volumes/kubernetes.io~projected/kube-api-access-9chtw:{mountpoint:/var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volumes/kubernetes.io~projected/kube-api-access-9chtw major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b52cb20e-db43-407c-bf36-870475ceadb7/volumes/kubernetes.io~projected/kube-api-access-8t4jt:{mountpoint:/var/lib/kubelet/pods/b52cb20e-db43-407c-bf36-870475ceadb7/volumes/kubernetes.io~projected/kube-api-access-8t4jt major:0 minor:74 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6/volumes/kubernetes.io~projected/kube-api-access-tmj9k:{mountpoint:/var/lib/kubelet/pods/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6/volumes/kubernetes.io~projected/kube-api-access-tmj9k major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd61c76f-9514-46db-99e2-21c6280a4842/volumes/kubernetes.io~projected/kube-api-access-nbjs2:{mountpoint:/var/lib/kubelet/pods/bd61c76f-9514-46db-99e2-21c6280a4842/volumes/kubernetes.io~projected/kube-api-access-nbjs2 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3952ef0-db1a-4c93-b9e4-fccda9618beb/volumes/kubernetes.io~projected/kube-api-access-r8xd4:{mountpoint:/var/lib/kubelet/pods/e3952ef0-db1a-4c93-b9e4-fccda9618beb/volumes/kubernetes.io~projected/kube-api-access-r8xd4 major:0 minor:75 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaf9b7e7-396a-454b-a7cc-94baedd50371/volumes/kubernetes.io~projected/kube-api-access-2hsz7:{mountpoint:/var/lib/kubelet/pods/eaf9b7e7-396a-454b-a7cc-94baedd50371/volumes/kubernetes.io~projected/kube-api-access-2hsz7 major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaf9b7e7-396a-454b-a7cc-94baedd50371/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/eaf9b7e7-396a-454b-a7cc-94baedd50371/volumes/kubernetes.io~secret/cert major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c/volumes/kubernetes.io~projected/kube-api-access-vp9fw:{mountpoint:/var/lib/kubelet/pods/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c/volumes/kubernetes.io~projected/kube-api-access-vp9fw major:0 minor:77 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb7c72f6-d102-47f7-a69b-d720dac0fd6b/volumes/kubernetes.io~projected/kube-api-access-2bd65:{mountpoint:/var/lib/kubelet/pods/fb7c72f6-d102-47f7-a69b-d720dac0fd6b/volumes/kubernetes.io~projected/kube-api-access-2bd65 major:0 minor:72 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-101:{mountpoint:/var/lib/containers/storage/overlay/eb9a90aa396c4b564ebdcdc367b8139c93acfdd5c06180b5841b36e739050378/merged major:0 minor:101 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/6db279d18f7ac5b3595fb9cad872022722708d10992b29d57694904688b08201/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/add112b17f4c9cdc52aaf2fa882345e1ba69741f9131e616e3465efdf574e7e7/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b4432701ec08fd62281436485f7434b796ab3c356da6dc69a0a10ef98d9c1b2c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/358a7fb213739548103f975424d575eb0d196225d23906ee528313ad4afb7e1b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fc318ae895a2b442c4aa56546d63462b9c0c06922f1c7aa2bbee35aa8e93601b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6b6736b799a8e898b5035802f2f7e381a7b0d5528633eb7cdec22bae6fcc6707/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/caab95a344030f340e1a18cff4c636bf0a3f2b17e60ccb791d7b83ff648911f7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/91f96a198d62bf58f7b47b9ceb991a9de33fe6e27829058d809ecf81c9d78d7e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/3a9133bfd153b309624f5493647a38d859fa447dce2eef94a08a42f2b15b999b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/46be9ec4ea7ad54a6080400f8b646c8625afa52a5b9e22dcaefeab2f0d9ebc9b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/8d7424dcbbe4383d020adb31e6b134c1cd763aab88930e42e0d32a99f6a29e5f/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/fdb79af0cf65c7068e9e86d933b6c50691f045276e778be8b9e51f533be784c6/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/c41ccaaf36886efbcd6a00fc3a4511ab90aa0081d4eed451e464cd13052eea67/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/e638fea364553a9d437e6d3ea6eb87a5d8f91f4c3ae3dd488f59c82031cd69ef/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/778ae88f2ccdbc70334832b87be8cf37ae01fc36b980502a1d6927563777c13e/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/cb8ac5f42694285dec7a9827a39d873032b9eb475ea5420b5e06d5bb3fedd26c/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/76239d2735cf5b928e845b2bd639d29c3c0b770535f1318ccb5c0b717caed3c8/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/f68e77b0b440151f0e906508c8c0ea0ca18f5d603ed3d6a75197a734b3936312/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/c895b7f99ba4ef2bf48b8f4b5fe3b1e8441f8f6a8409ff17d67a173f8d108a22/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/edbcc4cc54efcf02f9fb3afd2af665890ac7a5c56fd0d571be0aa183c55b29cb/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/4e34ba2b751030f9c4e69066c79fd5bd8c8d0e16526aaaebfe4fcda875f0d01a/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/6bf596a4104ffbdfbe3c51e0dcf426cf0fb67b9495bbccb371e28c8802cd9064/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/863dad10ac7fb016a2d250f06e137bf24cfec4ecb1aed9b419e30554ce70c0e7/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/9aa234f58216e441babf5a345eeb087a6929688d4988c064d60b79ffffd7d976/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/0d80f567793a96f7a349d5a43f0f93c4ea3de450add7520399c27f14606477dc/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/541295842d9587baaa34c9b1f69e30b6c4cf81c588a97c3b9c6837db183ba6e5/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/e40b8dda31e502d80c76005e2983e8a70e2e2385f7b4ee31c824249ca25bfac6/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/f637991d90c5d32ef264f6a49eee112c2f7b1bb878889e59fcfea8bea538e06b/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/f74012e2c9085570d1e32592afa9e8859706d5e5509be3c9409606a176f005af/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c6a0c7c96859066da4e711a40834f9695222b81bc41d48e3f942f315e3b403c2/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/b87fda87df0943d936e28bee6796a6054573a6ba27c8616853866f7a3a13fd78/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/b45a95f76aab9134a7be55b1e609deca3e5dfbdc18e4277c640087d08055f323/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/446ecc334d57f81394beda260c6ac78e587601a48dbed3cdd5f3b2352a8ee45b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f0204e350817f704ef2fc8cf523c6cdea9dc1a881a249782a557105fad35ecb2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/2c9362894f06abae04f448b8d0714dd8e16ebffb54b65061ae6b92be29a7d8db/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-59:{mountpoint:/var/lib/containers/storage/overlay/020dce49687b6c8b600dde76fdc66ce578c00398227ba7139cc02d30cc91fe13/merged major:0 minor:59 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/b141674ea00d12066a7ae217c8da5822686e4e675b2955cd9d4a2b2b80def3e9/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/612c3af61de0517127dbef801058f1b669d9212e2312f68b0c989a1dcfd7484c/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/27f814575d08c132f2660f588dad1815a0707253196a2cc1b0ef83d6167cac51/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/e8eb42781e5b36601a3f9e740c5c8fa3107fe30d9728d737b4b7fdc840bfc1fb/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/a59c3ad7a7d54062ee18604b04f674461b181a276da24ed276ef50a8b61ea879/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/3011d27e6e680553f5b9c0c98b0ced5e3f896cf6cd8ca37b87e2b3be216270b9/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.237899 4582 manager.go:217] Machine: {Timestamp:2026-10-30 17:58:23.237371259 +0000 UTC m=+0.061765543 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101596 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b3a7548125186f5d9763963d78dfb SystemUUID:ec2b3a75-4812-5186-f5d9-763963d78dfb BootID:7c3d372c-ac0e-4d9e-8931-432f273512d9 Filesystems:[{Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71ddb07e0b5f77310b67a3c582e4acff6545c9908588f9e1acffae511a57a597/userdata/shm DeviceMajor:0 DeviceMinor:55 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/347508d719c82b6df9a48faca4f859f6fe9dfa4279d07c6f6974c774567c7ea9/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-59 DeviceMajor:0 DeviceMinor:59 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b291d1d26b946b36ac46dd56c03a8d44d2fc8ad74dc57cc9cfe4de699030e930/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6/volumes/kubernetes.io~projected/kube-api-access-tmj9k DeviceMajor:0 DeviceMinor:69 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b52cb20e-db43-407c-bf36-870475ceadb7/volumes/kubernetes.io~projected/kube-api-access-8t4jt DeviceMajor:0 DeviceMinor:74 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eaf9b7e7-396a-454b-a7cc-94baedd50371/volumes/kubernetes.io~projected/kube-api-access-2hsz7 DeviceMajor:0 DeviceMinor:200 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0113161-09c9-4be8-9e2d-d9dd939b7593/volumes/kubernetes.io~projected/kube-api-access-tzh9v DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8eabc5fe986e84f49724075384d88321882b639a30b3d0a566571d5532e5de47/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91d20aa529a1f4818f6aab1ad0912685b8780c97b8a918abdee9eb59948a7fce/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:68 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/845bfa8a-e757-4cd4-bcd0-f3625bb21afa/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5df9464e-c388-44ff-a808-e69b6dcf7d4d/volumes/kubernetes.io~projected/kube-api-access-k7gxn DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022/volumes/kubernetes.io~projected/kube-api-access-4ljf9 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~projected/kube-api-access-ncnxb DeviceMajor:0 DeviceMinor:71 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2cf30b64-3606-4e62-894c-1c5fbb2e7190/volumes/kubernetes.io~projected/kube-api-access-g6hbk DeviceMajor:0 DeviceMinor:73 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0113161-09c9-4be8-9e2d-d9dd939b7593/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/485291dd2b86da2b799fc215672372822537cb410dec2cafdad694a0a126966d/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f9aaf7d9414300042204ff8db23cf3e02438971ff6f984d37cc916e5b0147f4/userdata/shm DeviceMajor:0 DeviceMinor:99 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd334a8-7dd5-4a51-8ffc-adb0c815682f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3952ef0-db1a-4c93-b9e4-fccda9618beb/volumes/kubernetes.io~projected/kube-api-access-r8xd4 DeviceMajor:0 DeviceMinor:75 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23fa282b57ed4364869fde9c5c71628742028abfd7982a22f947114a68929035/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44c98c0c8b57be286fbdd036ccf06de9be3c0cc1f4d25324d57f5606dfe5f0ec/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/282f89fc6274e9439fdc66059961674bd1fe310b79391587a40612bae9e2ffff/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5451f7b3cbd6c6946950be2175fbf01234994fd4dd4891f9404d19a83fd4a70b/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f7d3975-f96e-4f38-9535-1f37d2142493/volumes/kubernetes.io~projected/kube-api-access-h9l7f DeviceMajor:0 DeviceMinor:76 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f0329ae-a289-4958-83b9-ff0c5455c068/volumes/kubernetes.io~projected/kube-api-access-flvvz DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1298430fd8f01d571a0d5ea01f613516e727d85850b746bc88a1349dc62c2e86/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eaf9b7e7-396a-454b-a7cc-94baedd50371/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:196 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c/volumes/kubernetes.io~projected/kube-api-access-vp9fw DeviceMajor:0 DeviceMinor:77 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd61c76f-9514-46db-99e2-21c6280a4842/volumes/kubernetes.io~projected/kube-api-access-nbjs2 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4626f52e99f06ab0530b7b5c7ee4bf18ccc399305a7adbdb15da4a59c0117687/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1489b258-153a-47ac-8740-c4e9417b90c6/volumes/kubernetes.io~projected/kube-api-access-scbbb DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2daed88-3483-4e79-a6cc-f12cddc1bdac/volumes/kubernetes.io~projected/kube-api-access-9chtw DeviceMajor:0 DeviceMinor:70 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb7c72f6-d102-47f7-a69b-d720dac0fd6b/volumes/kubernetes.io~projected/kube-api-access-2bd65 DeviceMajor:0 DeviceMinor:72 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:282f89fc6274e94 MacAddress:1e:d4:60:03:01:a5 Speed:10000 Mtu:8901} {Name:347b895d1f06e3c MacAddress:e6:f2:d5:58:d6:eb Speed:10000 Mtu:8901} {Name:4626f52e99f06ab MacAddress:16:e2:01:fd:04:e3 Speed:10000 Mtu:8901} {Name:5451f7b3cbd6c69 MacAddress:ca:9b:9d:21:26:80 Speed:10000 Mtu:8901} {Name:8eabc5fe986e84f MacAddress:5a:7d:67:69:6c:cf Speed:10000 Mtu:8901} {Name:91d20aa529a1f48 MacAddress:72:59:53:43:d3:8f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fd:e8:9f:37 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:3e:96:54:c2:e7 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:fd:e8:9f:37 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0e:b4:8e:8d:33:26 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:20:5e:39:f0:27 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238147 4582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238187 4582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238470 4582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238488 4582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238617 4582 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238625 4582 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238637 4582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238657 4582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238745 4582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.238915 4582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239024 4582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239057 4582 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239067 4582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239078 4582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239086 4582 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239093 4582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239703 4582 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.239715 4582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.240165 4582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242172 4582 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242182 4582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242268 4582 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242460 4582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242492 4582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242581 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242594 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242600 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242606 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242612 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242618 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242624 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242630 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242638 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242643 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242656 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242665 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242705 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:58:23.245781 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.242711 4582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:58:23.248049 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.246023 4582 server.go:1263] "Started kubelet" Oct 30 17:58:23.248049 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.246169 4582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:58:23.248049 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.246148 4582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:58:23.248049 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.247681 4582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:58:23.248049 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.247987 4582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:58:23.246785 ip-10-0-133-124 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:58:23.248571 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.248557 4582 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:58:23.251950 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.251906 4582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:58:23.251950 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.251940 4582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.251994 4582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:45 +0000 UTC" deadline="2028-05-22 02:26:34.784753658 +0000 UTC" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.252066 4582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13664h28m11.53269133s" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.252070 4582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.252087 4582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.252159 4582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:23.252363 4582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-124.ec2.internal\" not found" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.253995 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0329ae-a289-4958-83b9-ff0c5455c068" volumeName="kubernetes.io/configmap/3f0329ae-a289-4958-83b9-ff0c5455c068-config-volume" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254049 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0329ae-a289-4958-83b9-ff0c5455c068" volumeName="kubernetes.io/projected/3f0329ae-a289-4958-83b9-ff0c5455c068-kube-api-access-flvvz" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254063 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd334a8-7dd5-4a51-8ffc-adb0c815682f" volumeName="kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-tmp" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254076 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f7d3975-f96e-4f38-9535-1f37d2142493" volumeName="kubernetes.io/empty-dir/6f7d3975-f96e-4f38-9535-1f37d2142493-tmp-dir" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254087 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" volumeName="kubernetes.io/projected/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-kube-api-access-tmj9k" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254099 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89b785d-9cd4-48ed-a4f7-85b5ddcd086c" volumeName="kubernetes.io/configmap/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-serviceca" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254110 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89b785d-9cd4-48ed-a4f7-85b5ddcd086c" volumeName="kubernetes.io/projected/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-kube-api-access-vp9fw" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254121 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaf9b7e7-396a-454b-a7cc-94baedd50371" volumeName="kubernetes.io/projected/eaf9b7e7-396a-454b-a7cc-94baedd50371-kube-api-access-2hsz7" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254135 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" volumeName="kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254146 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cf30b64-3606-4e62-894c-1c5fbb2e7190" volumeName="kubernetes.io/projected/2cf30b64-3606-4e62-894c-1c5fbb2e7190-kube-api-access-g6hbk" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254157 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0329ae-a289-4958-83b9-ff0c5455c068" volumeName="kubernetes.io/empty-dir/3f0329ae-a289-4958-83b9-ff0c5455c068-tmp-dir" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254169 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0113161-09c9-4be8-9e2d-d9dd939b7593" volumeName="kubernetes.io/projected/a0113161-09c9-4be8-9e2d-d9dd939b7593-kube-api-access-tzh9v" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254180 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0113161-09c9-4be8-9e2d-d9dd939b7593" volumeName="kubernetes.io/secret/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-key" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254192 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" volumeName="kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254206 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2daed88-3483-4e79-a6cc-f12cddc1bdac" volumeName="kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254217 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b52cb20e-db43-407c-bf36-870475ceadb7" volumeName="kubernetes.io/projected/b52cb20e-db43-407c-bf36-870475ceadb7-kube-api-access-8t4jt" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254228 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5df9464e-c388-44ff-a808-e69b6dcf7d4d" volumeName="kubernetes.io/projected/5df9464e-c388-44ff-a808-e69b6dcf7d4d-kube-api-access-k7gxn" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254240 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" volumeName="kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254265 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" volumeName="kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254277 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" volumeName="kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-binary-copy" seLinuxMountContext="" Oct 30 17:58:23.254285 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254306 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd61c76f-9514-46db-99e2-21c6280a4842" volumeName="kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254321 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3952ef0-db1a-4c93-b9e4-fccda9618beb" volumeName="kubernetes.io/configmap/e3952ef0-db1a-4c93-b9e4-fccda9618beb-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254333 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a" volumeName="kubernetes.io/configmap/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-konnectivity-ca" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254349 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a" volumeName="kubernetes.io/secret/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-agent-certs" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254361 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd334a8-7dd5-4a51-8ffc-adb0c815682f" volumeName="kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-tuned" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254375 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2daed88-3483-4e79-a6cc-f12cddc1bdac" volumeName="kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-env-overrides" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254387 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2daed88-3483-4e79-a6cc-f12cddc1bdac" volumeName="kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-config" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254400 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7c72f6-d102-47f7-a69b-d720dac0fd6b" volumeName="kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-daemon-config" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254419 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1489b258-153a-47ac-8740-c4e9417b90c6" volumeName="kubernetes.io/empty-dir/1489b258-153a-47ac-8740-c4e9417b90c6-data-volume" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254435 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd334a8-7dd5-4a51-8ffc-adb0c815682f" volumeName="kubernetes.io/projected/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-kube-api-access-ncnxb" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254447 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f7d3975-f96e-4f38-9535-1f37d2142493" volumeName="kubernetes.io/projected/6f7d3975-f96e-4f38-9535-1f37d2142493-kube-api-access-h9l7f" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254458 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="845bfa8a-e757-4cd4-bcd0-f3625bb21afa" volumeName="kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254475 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" volumeName="kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254505 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3952ef0-db1a-4c93-b9e4-fccda9618beb" volumeName="kubernetes.io/projected/e3952ef0-db1a-4c93-b9e4-fccda9618beb-kube-api-access-r8xd4" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254507 4582 factory.go:55] Registering systemd factory Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254517 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaf9b7e7-396a-454b-a7cc-94baedd50371" volumeName="kubernetes.io/secret/eaf9b7e7-396a-454b-a7cc-94baedd50371-cert" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254526 4582 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254530 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7c72f6-d102-47f7-a69b-d720dac0fd6b" volumeName="kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cni-binary-copy" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254541 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" volumeName="kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254552 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7c72f6-d102-47f7-a69b-d720dac0fd6b" volumeName="kubernetes.io/projected/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-kube-api-access-2bd65" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254563 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1489b258-153a-47ac-8740-c4e9417b90c6" volumeName="kubernetes.io/projected/1489b258-153a-47ac-8740-c4e9417b90c6-kube-api-access-scbbb" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254574 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0113161-09c9-4be8-9e2d-d9dd939b7593" volumeName="kubernetes.io/configmap/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-cabundle" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254584 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2daed88-3483-4e79-a6cc-f12cddc1bdac" volumeName="kubernetes.io/projected/a2daed88-3483-4e79-a6cc-f12cddc1bdac-kube-api-access-9chtw" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254597 4582 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2daed88-3483-4e79-a6cc-f12cddc1bdac" volumeName="kubernetes.io/secret/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254608 4582 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.254616 4582 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.255095 4582 factory.go:153] Registering CRI-O factory Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.255112 4582 factory.go:223] Registration of the crio container factory successfully Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.255172 4582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.255195 4582 factory.go:103] Registering Raw factory Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.255210 4582 manager.go:1199] Started watching for new ooms in manager Oct 30 17:58:23.256143 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.255535 4582 manager.go:319] Starting recovery of all containers Oct 30 17:58:23.290732 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.290684 4582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:58:23.292007 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.291984 4582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:58:23.292112 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.292012 4582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:58:23.292112 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.292034 4582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:58:23.292112 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:23.292074 4582 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:58:23.296373 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.296345 4582 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:58:23.298633 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.298077 4582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.298633 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.298200 4582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.307941 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.307918 4582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.313308 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.313290 4582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:23.315083 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.315019 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="a273ca0930ed8bb4c7f9056ee676ffe8d030033a5bbd04241bc02c3a69f0b849" exitCode=0 Oct 30 17:58:23.315083 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.315049 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="d5e39eaf50bb0e32298a0d6316ef611bf835501df6c956251c3d22facb444f87" exitCode=0 Oct 30 17:58:23.315083 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.315059 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="06f459c3ca91c117b589c216735ef06043de2a5ef90ae9f04556fe7c02a39888" exitCode=0 Oct 30 17:58:23.315083 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.315071 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="b089a5808d4e410d87738f02e797914de63cadced1e408e44c26baabef2316e9" exitCode=0 Oct 30 17:58:23.315355 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.315081 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="facd4d754f90dcdf7f1fc033b071ceab3a446a60b18f0d5d81f7be2dbc815b58" exitCode=0 Oct 30 17:58:23.315355 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.315107 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a9039c-06fd-4b76-adbd-92f5bb7e94b6" containerID="4e97187dc6fa69e4d11b9802aac18f3f3f0f39b8d9e6d6434b3b21e88ab8a56b" exitCode=0 Oct 30 17:58:23.326025 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.326000 4582 generic.go:356] "Generic (PLEG): container finished" podID="9108d1d7bcbaabf4aba6b0607ef8c386" containerID="2a00aa2391414eb1b75d1d9a6a793eae385d2d7f3a6912082486ed8c8783f2c7" exitCode=0 Oct 30 17:58:23.332969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.332949 4582 manager.go:324] Recovery completed Oct 30 17:58:23.348046 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.347750 4582 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:58:23.348046 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.347771 4582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:58:23.348046 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.347793 4582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:58:23.348046 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348020 4582 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:58:23.348305 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348036 4582 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:58:23.348305 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348063 4582 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:58:23.348305 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348072 4582 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:58:23.348305 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348079 4582 policy_none.go:50] "Start" Oct 30 17:58:23.348305 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348097 4582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:58:23.348305 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348109 4582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:58:23.348546 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348373 4582 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:58:23.348546 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348384 4582 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:58:23.348546 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.348393 4582 policy_none.go:44] "Start" Oct 30 17:58:23.353448 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.353418 4582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:58:23.356271 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.356259 4582 manager.go:341] "Starting Device Plugin manager" Oct 30 17:58:23.357114 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:23.356286 4582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:58:23.357175 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357119 4582 server.go:87] "Starting device plugin registration server" Oct 30 17:58:23.357471 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357459 4582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:58:23.357503 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357475 4582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:58:23.357593 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357573 4582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:58:23.357731 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357695 4582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:58:23.357786 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357745 4582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:58:23.357786 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357752 4582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:58:23.357910 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.357794 4582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:58:23.357729791Z","UUID":"4c1c5596-8c1a-4ae9-97f5-e92701a5906e","Handler":null,"Name":"","Endpoint":""} Oct 30 17:58:23.359129 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:23.359106 4582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:58:23.359273 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.359256 4582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:58:23.359353 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.359281 4582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:58:23.393134 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393094 4582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal"] Oct 30 17:58:23.393610 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393572 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerStarted","Data":"e1fb641f70ce6910f4630792708544220d56294f087b9c42176d34af3974ca9b"} Oct 30 17:58:23.393610 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393607 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerDied","Data":"2a00aa2391414eb1b75d1d9a6a793eae385d2d7f3a6912082486ed8c8783f2c7"} Oct 30 17:58:23.393755 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393621 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" event={"ID":"9108d1d7bcbaabf4aba6b0607ef8c386","Type":"ContainerStarted","Data":"e818fcd12418847d38f0792b256dbebec0c313802b31c389d9af718130432f16"} Oct 30 17:58:23.393755 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393639 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" event={"ID":"2c67125f85ae263b41b821237802c161","Type":"ContainerStarted","Data":"dbd3a9173fe1dc01879b9834e9a35829fc8644635f4e8b3cce3f9b80a92233a0"} Oct 30 17:58:23.393755 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393650 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" event={"ID":"2c67125f85ae263b41b821237802c161","Type":"ContainerStarted","Data":"a8a40aba98a74c2a260bb608c109b219158bf7d40795c4a0e418ec99cba4c87a"} Oct 30 17:58:23.393755 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393683 4582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.393755 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.393741 4582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.417244 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.417222 4582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:23.417381 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:23.417276 4582 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.418269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.418148 4582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:23.418332 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:23.418297 4582 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.458386 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.458332 4582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:58:23.459679 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.459651 4582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:58:23.459768 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.459683 4582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:58:23.459768 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.459697 4582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-124.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:58:23.459768 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.459730 4582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.489791 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.489768 4582 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.489898 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.489846 4582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.497830 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.497793 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.497932 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.497843 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.497932 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.497870 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c67125f85ae263b41b821237802c161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" (UID: \"2c67125f85ae263b41b821237802c161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.598458 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.598430 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.598458 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.598448 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.598586 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.598469 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.598586 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.598487 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c67125f85ae263b41b821237802c161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" (UID: \"2c67125f85ae263b41b821237802c161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.598586 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.598533 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c67125f85ae263b41b821237802c161-config\") pod \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" (UID: \"2c67125f85ae263b41b821237802c161\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:58:23.598586 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:23.598559 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9108d1d7bcbaabf4aba6b0607ef8c386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" (UID: \"9108d1d7bcbaabf4aba6b0607ef8c386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:24.239973 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.239938 4582 apiserver.go:52] "Watching apiserver" Oct 30 17:58:24.283170 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.283148 4582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:58:24.283587 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.283568 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ckrtw","openshift-multus/network-metrics-daemon-prjhj","openshift-cluster-node-tuning-operator/tuned-d65dr","openshift-dns/dns-default-mmstz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n","openshift-ingress-canary/ingress-canary-ml8bz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal","openshift-network-operator/iptables-alerter-c5z8z","openshift-dns/node-resolver-jrtkt","openshift-insights/insights-runtime-extractor-4d6r5","openshift-network-diagnostics/network-check-target-c5qcf","openshift-insights/periodic-gathering-hn6rb-55v8r","openshift-ovn-kubernetes/ovnkube-node-rb89h","kube-system/global-pull-secret-syncer-jg28z","openshift-service-ca/service-ca-5fffc97bdf-csb7v","openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg","openshift-multus/multus-additional-cni-plugins-86gbr","kube-system/konnectivity-agent-5jtxd","kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal","openshift-image-registry/node-ca-rfjbn"] Oct 30 17:58:24.284561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.284513 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:24.288935 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.288915 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.293529 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.293511 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.309188 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.309167 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:58:24.309281 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.309225 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.309281 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.309252 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:58:24.309394 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.309289 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:58:24.309394 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.309352 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh48h\"" Oct 30 17:58:24.309472 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.309409 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6fpb9\"" Oct 30 17:58:24.333590 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.333571 4582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:58:24.333745 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.333731 4582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:24.351206 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.351181 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:24.354216 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.354203 4582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:24.354266 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:24.354247 4582 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-124.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-124.ec2.internal" Oct 30 17:58:24.354711 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.354697 4582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:58:24.354764 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.354713 4582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:24.354764 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:24.354743 4582 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-124.ec2.internal" Oct 30 17:58:24.368518 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.368448 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.368518 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.368475 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.368518 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.368490 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.368640 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.368525 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.368640 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.368556 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.369797 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.369780 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.369902 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.369917 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.369908 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.369973 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.369983 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370001 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370005 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370004 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370015 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370035 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370069 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370048 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370149 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370214 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370270 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370343 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370386 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370402 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370414 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370440 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370442 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370461 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370457 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370470 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370494 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:24.370885 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370496 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:58:24.370885 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370511 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370885 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370558 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:58:24.370885 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.370593 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:58:24.379939 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.379912 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:58:24.380081 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.380029 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:58:24.402965 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.402941 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.403058 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.402983 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4jt\" (UniqueName: \"kubernetes.io/projected/b52cb20e-db43-407c-bf36-870475ceadb7-kube-api-access-8t4jt\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.403058 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403012 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.403058 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403041 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-systemd\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.403212 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403066 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-tmp\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.403212 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403097 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.403212 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403164 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7d3975-f96e-4f38-9535-1f37d2142493-hosts-file\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:58:24.403212 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403198 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-slash\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.403395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403232 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-systemd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.403395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403261 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-node-log\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.403395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403288 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-socket-dir-parent\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.403395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403329 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:58:24.403395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403315 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-kubelet\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.403395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403100 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.403651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403343 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt2qr\"" Oct 30 17:58:24.403651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403431 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-tuned\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.403651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403504 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.403651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403539 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-tuned\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.403651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403604 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-tmp\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.403651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403590 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-ovn\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.403651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403618 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzq26\"" Oct 30 17:58:24.403988 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403728 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:58:24.403988 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403933 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.404082 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403995 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:58:24.404082 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.403950 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7dzl7\"" Oct 30 17:58:24.404082 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404010 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-bin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.404082 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404036 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.404269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404060 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.404269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404036 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.404269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404145 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.404269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404240 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:58:24.404439 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404324 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-conf\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.404439 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404405 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-run\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.404528 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404378 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.404528 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404479 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2xlsx\"" Oct 30 17:58:24.404528 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404491 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-registration-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.404661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404532 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wj5nd\"" Oct 30 17:58:24.404661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404559 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hbk\" (UniqueName: \"kubernetes.io/projected/2cf30b64-3606-4e62-894c-1c5fbb2e7190-kube-api-access-g6hbk\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:24.404661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404596 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9m88r\"" Oct 30 17:58:24.404661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404598 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-host\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.404661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404623 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8mdkh\"" Oct 30 17:58:24.404935 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404662 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:58:24.404981 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404924 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.405029 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.404998 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3952ef0-db1a-4c93-b9e4-fccda9618beb-iptables-alerter-script\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:58:24.405077 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405045 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t4kbs\"" Oct 30 17:58:24.405077 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405043 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7d3975-f96e-4f38-9535-1f37d2142493-tmp-dir\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:58:24.405167 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405086 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-kubelet-config\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405323 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405344 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405360 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-cabundle\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405364 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6627\"" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405403 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xl6hf\"" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405444 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-kubelet\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405477 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-var-lib-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405491 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7d3975-f96e-4f38-9535-1f37d2142493-tmp-dir\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405447 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p48xz\"" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405512 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chtw\" (UniqueName: \"kubernetes.io/projected/a2daed88-3483-4e79-a6cc-f12cddc1bdac-kube-api-access-9chtw\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405503 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3952ef0-db1a-4c93-b9e4-fccda9618beb-iptables-alerter-script\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405530 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nwsqb\"" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405532 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdnw2\"" Oct 30 17:58:24.405550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405552 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-socket-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405599 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-swnsr\"" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405620 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-cabundle\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405728 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mngtd\"" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405729 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405778 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-sys-fs\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405890 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405940 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.405920 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-serviceca\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406004 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-k8s-cni-cncf-io\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406030 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbbb\" (UniqueName: \"kubernetes.io/projected/1489b258-153a-47ac-8740-c4e9417b90c6-kube-api-access-scbbb\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406058 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406104 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-bin\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.406147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406144 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-env-overrides\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406173 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-multus-certs\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406222 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406253 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-serviceca\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406260 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-device-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406311 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1489b258-153a-47ac-8740-c4e9417b90c6-crio-socket\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406338 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysconfig\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406343 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jcq9r\"" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406391 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-system-cni-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406461 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-multus\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406478 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-env-overrides\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406505 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljf9\" (UniqueName: \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406536 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406562 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-kubernetes\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406615 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-dbus\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.406657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406650 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-log-socket\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.407262 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406676 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-system-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.407262 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406707 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-conf-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.407262 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.406747 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsz7\" (UniqueName: \"kubernetes.io/projected/eaf9b7e7-396a-454b-a7cc-94baedd50371-kube-api-access-2hsz7\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:24.407262 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407133 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.407262 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407142 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.407262 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407216 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-agent-certs\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:58:24.407262 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407250 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3952ef0-db1a-4c93-b9e4-fccda9618beb-host-slash\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:58:24.407543 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407292 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvvz\" (UniqueName: \"kubernetes.io/projected/3f0329ae-a289-4958-83b9-ff0c5455c068-kube-api-access-flvvz\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.407621 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407603 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzh9v\" (UniqueName: \"kubernetes.io/projected/a0113161-09c9-4be8-9e2d-d9dd939b7593-kube-api-access-tzh9v\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:24.407683 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407660 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-script-lib\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.407735 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407696 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cnibin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407879 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-netns\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407930 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-modprobe-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.407959 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmj9k\" (UniqueName: \"kubernetes.io/projected/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-kube-api-access-tmj9k\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408043 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-konnectivity-ca\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408096 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-netns\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408113 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-agent-certs\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408172 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-config\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408193 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-script-lib\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.408312 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408245 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-host\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408351 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a-konnectivity-ca\") pod \"konnectivity-agent-5jtxd\" (UID: \"e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a\") " pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408395 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-lib-modules\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408433 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovnkube-config\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408440 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-var-lib-kubelet\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408502 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f0329ae-a289-4958-83b9-ff0c5455c068-tmp-dir\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408557 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-etc-kubernetes\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408592 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b7e7-396a-454b-a7cc-94baedd50371-cert\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408657 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnxb\" (UniqueName: \"kubernetes.io/projected/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-kube-api-access-ncnxb\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408701 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xd4\" (UniqueName: \"kubernetes.io/projected/e3952ef0-db1a-4c93-b9e4-fccda9618beb-kube-api-access-r8xd4\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408736 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.408782 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408774 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0329ae-a289-4958-83b9-ff0c5455c068-config-volume\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408885 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-systemd-units\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408915 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-netd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408950 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1489b258-153a-47ac-8740-c4e9417b90c6-data-volume\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.408984 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409015 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-sys\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409049 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-key\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409075 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cnibin\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409109 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9l7f\" (UniqueName: \"kubernetes.io/projected/6f7d3975-f96e-4f38-9535-1f37d2142493-kube-api-access-h9l7f\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409226 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409255 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-etc-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409289 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovn-node-metrics-cert\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.409327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409321 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409353 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cni-binary-copy\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409389 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-hostroot\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409418 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-etc-selinux\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409451 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409486 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gxn\" (UniqueName: \"kubernetes.io/projected/5df9464e-c388-44ff-a808-e69b6dcf7d4d-kube-api-access-k7gxn\") pod \"migrator-66df5d496b-whsrg\" (UID: \"5df9464e-c388-44ff-a808-e69b6dcf7d4d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409518 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409558 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9fw\" (UniqueName: \"kubernetes.io/projected/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-kube-api-access-vp9fw\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409591 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-daemon-config\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.409887 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.409619 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bd65\" (UniqueName: \"kubernetes.io/projected/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-kube-api-access-2bd65\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.411581 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.410334 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-daemon-config\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.411581 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.410710 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0329ae-a289-4958-83b9-ff0c5455c068-config-volume\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.411581 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.410934 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eaf9b7e7-396a-454b-a7cc-94baedd50371-cert\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:24.411581 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.411000 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f0329ae-a289-4958-83b9-ff0c5455c068-tmp-dir\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.411772 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.411682 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1489b258-153a-47ac-8740-c4e9417b90c6-data-volume\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.411772 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.411731 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0113161-09c9-4be8-9e2d-d9dd939b7593-signing-key\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:24.412049 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.412009 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-original-pull-secret\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.412410 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.412388 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2daed88-3483-4e79-a6cc-f12cddc1bdac-ovn-node-metrics-cert\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.412496 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.412477 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cni-binary-copy\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.419969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.419948 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4jt\" (UniqueName: \"kubernetes.io/projected/b52cb20e-db43-407c-bf36-870475ceadb7-kube-api-access-8t4jt\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.420541 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.420522 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hbk\" (UniqueName: \"kubernetes.io/projected/2cf30b64-3606-4e62-894c-1c5fbb2e7190-kube-api-access-g6hbk\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:24.421071 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.421052 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chtw\" (UniqueName: \"kubernetes.io/projected/a2daed88-3483-4e79-a6cc-f12cddc1bdac-kube-api-access-9chtw\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.422425 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.422406 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbbb\" (UniqueName: \"kubernetes.io/projected/1489b258-153a-47ac-8740-c4e9417b90c6-kube-api-access-scbbb\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.422584 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.422569 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvvz\" (UniqueName: \"kubernetes.io/projected/3f0329ae-a289-4958-83b9-ff0c5455c068-kube-api-access-flvvz\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.422676 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.422659 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljf9\" (UniqueName: \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\") pod \"periodic-gathering-hn6rb-55v8r\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:58:24.424969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.424954 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsz7\" (UniqueName: \"kubernetes.io/projected/eaf9b7e7-396a-454b-a7cc-94baedd50371-kube-api-access-2hsz7\") pod \"ingress-canary-ml8bz\" (UID: \"eaf9b7e7-396a-454b-a7cc-94baedd50371\") " pod="openshift-ingress-canary/ingress-canary-ml8bz" Oct 30 17:58:24.427664 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.427636 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzh9v\" (UniqueName: \"kubernetes.io/projected/a0113161-09c9-4be8-9e2d-d9dd939b7593-kube-api-access-tzh9v\") pod \"service-ca-5fffc97bdf-csb7v\" (UID: \"a0113161-09c9-4be8-9e2d-d9dd939b7593\") " pod="openshift-service-ca/service-ca-5fffc97bdf-csb7v" Oct 30 17:58:24.428398 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.428378 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bd65\" (UniqueName: \"kubernetes.io/projected/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-kube-api-access-2bd65\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.428460 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.428378 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmj9k\" (UniqueName: \"kubernetes.io/projected/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-kube-api-access-tmj9k\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.428903 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.428888 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gxn\" (UniqueName: \"kubernetes.io/projected/5df9464e-c388-44ff-a808-e69b6dcf7d4d-kube-api-access-k7gxn\") pod \"migrator-66df5d496b-whsrg\" (UID: \"5df9464e-c388-44ff-a808-e69b6dcf7d4d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whsrg" Oct 30 17:58:24.429018 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.428999 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xd4\" (UniqueName: \"kubernetes.io/projected/e3952ef0-db1a-4c93-b9e4-fccda9618beb-kube-api-access-r8xd4\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:58:24.429184 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.429165 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnxb\" (UniqueName: \"kubernetes.io/projected/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-kube-api-access-ncnxb\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.429263 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.429249 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9l7f\" (UniqueName: \"kubernetes.io/projected/6f7d3975-f96e-4f38-9535-1f37d2142493-kube-api-access-h9l7f\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:58:24.429520 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.429499 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9fw\" (UniqueName: \"kubernetes.io/projected/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-kube-api-access-vp9fw\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:58:24.429669 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.429654 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjs2\" (UniqueName: \"kubernetes.io/projected/bd61c76f-9514-46db-99e2-21c6280a4842-kube-api-access-nbjs2\") pod \"network-check-target-c5qcf\" (UID: \"bd61c76f-9514-46db-99e2-21c6280a4842\") " pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:24.510368 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510341 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cnibin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.510474 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510379 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-netns\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.510474 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510404 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-modprobe-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.510474 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510428 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-netns\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510474 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510451 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-host\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:58:24.510474 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510454 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-cnibin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.510474 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510472 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-lib-modules\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510454 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-netns\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510496 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-var-lib-kubelet\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510522 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-etc-kubernetes\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510535 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-netns\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510546 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510551 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-var-lib-kubelet\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510539 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89b785d-9cd4-48ed-a4f7-85b5ddcd086c-host\") pod \"node-ca-rfjbn\" (UID: \"f89b785d-9cd4-48ed-a4f7-85b5ddcd086c\") " pod="openshift-image-registry/node-ca-rfjbn" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510564 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-modprobe-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510572 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-systemd-units\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510594 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-etc-kubernetes\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510625 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510653 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-netd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510690 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-systemd-units\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510712 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-netd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510716 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-lib-modules\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.510737 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510734 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510760 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-sys\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510793 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cnibin\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510832 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-etc-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510837 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-sys\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510867 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510880 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-etc-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510889 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-cnibin\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510893 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-hostroot\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:24.510884 4582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510929 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-hostroot\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510934 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-etc-selinux\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510943 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510966 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-etc-selinux\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:24.510977 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.010956603 +0000 UTC m=+1.835350902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : secret "metrics-daemon-secret" not found Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.510990 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511012 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511035 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-systemd\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511053 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-run-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511059 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7d3975-f96e-4f38-9535-1f37d2142493-hosts-file\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511088 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-slash\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511090 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-systemd\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511103 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f7d3975-f96e-4f38-9535-1f37d2142493-hosts-file\") pod \"node-resolver-jrtkt\" (UID: \"6f7d3975-f96e-4f38-9535-1f37d2142493\") " pod="openshift-dns/node-resolver-jrtkt" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511112 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-systemd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511099 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511130 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-slash\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511154 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-node-log\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511134 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-node-log\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511165 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-systemd\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511179 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-socket-dir-parent\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511208 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-kubelet\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511223 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-ovn\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511242 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511262 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-socket-dir-parent\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511270 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-kubelet\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511279 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-bin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511284 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511311 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-bin\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511319 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-run-ovn\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511333 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511363 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-conf\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511378 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-run\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511457 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-registration-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511471 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-host\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.511487 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511495 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-kubelet-config\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511521 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-kubelet\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511513 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-run\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511543 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-registration-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511522 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-conf\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511549 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-var-lib-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511581 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-kubelet\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511560 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-host\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511601 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-kubelet-config\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511591 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-socket-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511618 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-var-lib-openvswitch\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511642 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-sys-fs\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511648 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-socket-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511685 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511725 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-k8s-cni-cncf-io\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511748 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511687 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-sys-fs\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511772 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-bin\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511795 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-multus-certs\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511836 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-device-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511848 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-host-cni-bin\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511795 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-k8s-cni-cncf-io\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511881 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-run-multus-certs\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511882 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1489b258-153a-47ac-8740-c4e9417b90c6-crio-socket\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511902 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1489b258-153a-47ac-8740-c4e9417b90c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511888 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysctl-d\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511923 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b52cb20e-db43-407c-bf36-870475ceadb7-device-dir\") pod \"aws-ebs-csi-driver-node-drg7n\" (UID: \"b52cb20e-db43-407c-bf36-870475ceadb7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drg7n" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511927 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1489b258-153a-47ac-8740-c4e9417b90c6-crio-socket\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511931 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysconfig\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511958 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-system-cni-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511974 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-sysconfig\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.511985 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-multus\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512009 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512026 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-system-cni-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512033 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-host-var-lib-cni-multus\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512035 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-kubernetes\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512079 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd334a8-7dd5-4a51-8ffc-adb0c815682f-etc-kubernetes\") pod \"tuned-d65dr\" (UID: \"5dd334a8-7dd5-4a51-8ffc-adb0c815682f\") " pod="openshift-cluster-node-tuning-operator/tuned-d65dr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512081 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-dbus\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:24.512105 4582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512111 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-log-socket\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512135 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-system-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512139 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/845bfa8a-e757-4cd4-bcd0-f3625bb21afa-dbus\") pod \"global-pull-secret-syncer-jg28z\" (UID: \"845bfa8a-e757-4cd4-bcd0-f3625bb21afa\") " pod="kube-system/global-pull-secret-syncer-jg28z" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512168 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-conf-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512189 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a2daed88-3483-4e79-a6cc-f12cddc1bdac-log-socket\") pod \"ovnkube-node-rb89h\" (UID: \"a2daed88-3483-4e79-a6cc-f12cddc1bdac\") " pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512200 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512217 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-system-cni-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512233 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3952ef0-db1a-4c93-b9e4-fccda9618beb-host-slash\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512240 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb7c72f6-d102-47f7-a69b-d720dac0fd6b-multus-conf-dir\") pod \"multus-ckrtw\" (UID: \"fb7c72f6-d102-47f7-a69b-d720dac0fd6b\") " pod="openshift-multus/multus-ckrtw" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512239 4582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:24.512259 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.01222848 +0000 UTC m=+1.836622767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512277 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3952ef0-db1a-4c93-b9e4-fccda9618beb-host-slash\") pod \"iptables-alerter-c5z8z\" (UID: \"e3952ef0-db1a-4c93-b9e4-fccda9618beb\") " pod="openshift-network-operator/iptables-alerter-c5z8z" Oct 30 17:58:24.513383 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.512290 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a9039c-06fd-4b76-adbd-92f5bb7e94b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86gbr\" (UID: \"b5a9039c-06fd-4b76-adbd-92f5bb7e94b6\") " pod="openshift-multus/multus-additional-cni-plugins-86gbr" Oct 30 17:58:24.515556 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.514517 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0329ae-a289-4958-83b9-ff0c5455c068-metrics-tls\") pod \"dns-default-mmstz\" (UID: \"3f0329ae-a289-4958-83b9-ff0c5455c068\") " pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.519740 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.519717 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:24.590173 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.590153 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:24.884933 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.884906 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmstz"] Oct 30 17:58:24.891027 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:24.890994 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0329ae_a289_4958_83b9_ff0c5455c068.slice/crio-b57bb19305027f90b73d5c3252b1a8038c7b76361ae2eec5557fe155ba64c465 WatchSource:0}: Error finding container b57bb19305027f90b73d5c3252b1a8038c7b76361ae2eec5557fe155ba64c465: Status 404 returned error can't find the container with id b57bb19305027f90b73d5c3252b1a8038c7b76361ae2eec5557fe155ba64c465 Oct 30 17:58:24.893289 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.893266 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:58:24.920039 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:24.920009 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:25.015467 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:25.015441 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:25.015594 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:25.015492 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:25.015594 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:25.015581 4582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:25.015660 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:25.015629 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.015617295 +0000 UTC m=+2.840011564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:25.015660 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:25.015583 4582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:25.015721 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:25.015708 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.015694307 +0000 UTC m=+2.840088612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : secret "metrics-daemon-secret" not found Oct 30 17:58:25.111170 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:25.111140 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:25.336940 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:25.336865 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmstz" event={"ID":"3f0329ae-a289-4958-83b9-ff0c5455c068","Type":"ContainerStarted","Data":"b57bb19305027f90b73d5c3252b1a8038c7b76361ae2eec5557fe155ba64c465"} Oct 30 17:58:25.451448 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:25.451417 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rfjbn_f89b785d-9cd4-48ed-a4f7-85b5ddcd086c/node-ca/0.log" Oct 30 17:58:25.715926 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:25.715887 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:26.023347 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.023270 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:26.023347 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.023317 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:26.023583 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:26.023449 4582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:26.023583 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:26.023477 4582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:26.023583 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:26.023523 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:28.023507315 +0000 UTC m=+4.847901585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : secret "metrics-daemon-secret" not found Oct 30 17:58:26.023583 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:26.023541 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:28.023531165 +0000 UTC m=+4.847925438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:26.113979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.113944 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:26.197406 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.197362 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:26.251017 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.250986 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ml8bz_eaf9b7e7-396a-454b-a7cc-94baedd50371/serve-healthcheck-canary/0.log" Oct 30 17:58:26.375008 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.374985 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:26.555858 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.555813 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:26.716391 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.716319 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:26.851232 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:26.851204 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-whsrg_5df9464e-c388-44ff-a808-e69b6dcf7d4d/migrator/0.log" Oct 30 17:58:27.313410 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:27.313379 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:27.344620 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:27.344589 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmstz" event={"ID":"3f0329ae-a289-4958-83b9-ff0c5455c068","Type":"ContainerStarted","Data":"c890cf67599b820e4d53537778f1d1c407281f07d4d7e708aff586b91ed7e7e5"} Oct 30 17:58:27.344766 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:27.344643 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmstz" event={"ID":"3f0329ae-a289-4958-83b9-ff0c5455c068","Type":"ContainerStarted","Data":"229bee28e147c6261ff671ee9d31655943c9f2879f4ca8846837b65e4c45b526"} Oct 30 17:58:27.344766 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:27.344729 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:27.912102 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:27.912070 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:28.038377 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.038334 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:28.038574 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.038393 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:28.038574 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:28.038484 4582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:28.038574 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:28.038515 4582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:28.038574 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:28.038551 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:32.038536707 +0000 UTC m=+8.862930981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:28.038574 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:28.038567 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs podName:2cf30b64-3606-4e62-894c-1c5fbb2e7190 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:32.038560306 +0000 UTC m=+8.862954576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs") pod "network-metrics-daemon-prjhj" (UID: "2cf30b64-3606-4e62-894c-1c5fbb2e7190") : secret "metrics-daemon-secret" not found Oct 30 17:58:28.116100 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.116069 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:28.128399 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.128379 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:28.128607 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.128594 4582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:28.200204 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.200139 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:28.410295 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.410265 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:28.514227 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.514153 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:28.713946 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.713914 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:28.814522 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:28.814453 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:29.116844 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:29.116745 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:29.887032 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:29.887000 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:29.996657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:29.996627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:30.348554 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:30.348471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:30.444548 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:30.444516 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:30.545778 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:30.545746 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:30.710098 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:30.710068 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:31.128146 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:31.128113 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:31.960096 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:31.960063 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:32.063278 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.063243 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:32.063463 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.063290 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:32.063463 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:32.063395 4582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:32.063553 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:58:32.063462 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls podName:1489b258-153a-47ac-8740-c4e9417b90c6 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:40.06344745 +0000 UTC m=+16.887841723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d6r5" (UID: "1489b258-153a-47ac-8740-c4e9417b90c6") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:32.065475 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.065456 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf30b64-3606-4e62-894c-1c5fbb2e7190-metrics-certs\") pod \"network-metrics-daemon-prjhj\" (UID: \"2cf30b64-3606-4e62-894c-1c5fbb2e7190\") " pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:32.085699 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.085680 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prjhj" Oct 30 17:58:32.278011 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.277912 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:32.378416 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.378387 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prjhj"] Oct 30 17:58:32.383719 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:32.383685 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf30b64_3606_4e62_894c_1c5fbb2e7190.slice/crio-503871bfc4dc35d737740fc871e5b9603cc8590c926b711bf24840cd5ce8e6e2 WatchSource:0}: Error finding container 503871bfc4dc35d737740fc871e5b9603cc8590c926b711bf24840cd5ce8e6e2: Status 404 returned error can't find the container with id 503871bfc4dc35d737740fc871e5b9603cc8590c926b711bf24840cd5ce8e6e2 Oct 30 17:58:32.422141 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.422112 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:32.567633 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.567560 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5jtxd" Oct 30 17:58:32.591793 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.591748 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:32.752111 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.752079 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:32.863207 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.863110 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:32.963624 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:32.963582 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:33.324456 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:33.324430 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:33.368225 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:33.368194 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prjhj" event={"ID":"2cf30b64-3606-4e62-894c-1c5fbb2e7190","Type":"ContainerStarted","Data":"a878c09b66f5ba6de3585bd3a8382fb910471719c147b2c34b570683651f381d"} Oct 30 17:58:33.368345 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:33.368248 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prjhj" event={"ID":"2cf30b64-3606-4e62-894c-1c5fbb2e7190","Type":"ContainerStarted","Data":"503871bfc4dc35d737740fc871e5b9603cc8590c926b711bf24840cd5ce8e6e2"} Oct 30 17:58:33.794259 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:33.794222 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:33.896058 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:33.896023 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:34.372527 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:34.372485 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prjhj" event={"ID":"2cf30b64-3606-4e62-894c-1c5fbb2e7190","Type":"ContainerStarted","Data":"7c19c9d25754020f2fa9c564b90641974cf649a2b936e54cc9da1a019ca23013"} Oct 30 17:58:35.750375 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:35.750334 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:35.849924 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:35.849893 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:35.950207 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:35.950172 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:36.061718 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:36.061643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:36.282751 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:36.282719 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:37.121311 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:37.121284 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:37.210035 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:37.210006 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:37.352524 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:37.352497 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mmstz" Oct 30 17:58:38.236674 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.236641 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4"] Oct 30 17:58:38.238909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.238887 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" Oct 30 17:58:38.246827 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.246799 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-2g2gm"] Oct 30 17:58:38.248571 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.248557 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.255748 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.255729 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s9gbn\"" Oct 30 17:58:38.260970 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.260951 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4"] Oct 30 17:58:38.262014 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.261993 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 17:58:38.268016 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.267997 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2g2gm"] Oct 30 17:58:38.269751 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.269724 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:58:38.269853 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.269774 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:58:38.361291 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.361267 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5s4zd\"" Oct 30 17:58:38.376561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.376530 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:38.407076 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.407045 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f0119471-101c-491b-8a67-ca0c104d2261-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vpfj4\" (UID: \"f0119471-101c-491b-8a67-ca0c104d2261\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" Oct 30 17:58:38.407208 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.407112 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c78f02-e9ed-4344-b8fc-9a20c413548d-tmp\") pod \"downloads-6dd789c8b7-2g2gm\" (UID: \"24c78f02-e9ed-4344-b8fc-9a20c413548d\") " pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.407269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.407236 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc4k\" (UniqueName: \"kubernetes.io/projected/24c78f02-e9ed-4344-b8fc-9a20c413548d-kube-api-access-vjc4k\") pod \"downloads-6dd789c8b7-2g2gm\" (UID: \"24c78f02-e9ed-4344-b8fc-9a20c413548d\") " pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.508616 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.508515 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c78f02-e9ed-4344-b8fc-9a20c413548d-tmp\") pod \"downloads-6dd789c8b7-2g2gm\" (UID: \"24c78f02-e9ed-4344-b8fc-9a20c413548d\") " pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.508616 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.508570 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc4k\" (UniqueName: \"kubernetes.io/projected/24c78f02-e9ed-4344-b8fc-9a20c413548d-kube-api-access-vjc4k\") pod \"downloads-6dd789c8b7-2g2gm\" (UID: \"24c78f02-e9ed-4344-b8fc-9a20c413548d\") " pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.508883 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.508682 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f0119471-101c-491b-8a67-ca0c104d2261-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vpfj4\" (UID: \"f0119471-101c-491b-8a67-ca0c104d2261\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" Oct 30 17:58:38.509013 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.508991 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c78f02-e9ed-4344-b8fc-9a20c413548d-tmp\") pod \"downloads-6dd789c8b7-2g2gm\" (UID: \"24c78f02-e9ed-4344-b8fc-9a20c413548d\") " pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.511163 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.511133 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f0119471-101c-491b-8a67-ca0c104d2261-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vpfj4\" (UID: \"f0119471-101c-491b-8a67-ca0c104d2261\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" Oct 30 17:58:38.527909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.527885 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc4k\" (UniqueName: \"kubernetes.io/projected/24c78f02-e9ed-4344-b8fc-9a20c413548d-kube-api-access-vjc4k\") pod \"downloads-6dd789c8b7-2g2gm\" (UID: \"24c78f02-e9ed-4344-b8fc-9a20c413548d\") " pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.548883 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.548864 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" Oct 30 17:58:38.557768 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.557746 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:38.568461 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.568438 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:38.672360 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.672332 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:38.841917 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.841847 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4"] Oct 30 17:58:38.851732 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:38.851705 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0119471_101c_491b_8a67_ca0c104d2261.slice/crio-001950cb84bb555b299fed63c424b8c19c193532e4038eb40a7c2f6b322dd304 WatchSource:0}: Error finding container 001950cb84bb555b299fed63c424b8c19c193532e4038eb40a7c2f6b322dd304: Status 404 returned error can't find the container with id 001950cb84bb555b299fed63c424b8c19c193532e4038eb40a7c2f6b322dd304 Oct 30 17:58:38.863301 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:38.863279 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2g2gm"] Oct 30 17:58:38.868792 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:38.868771 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c78f02_e9ed_4344_b8fc_9a20c413548d.slice/crio-9fbb5f3ffb8f74e77d89907e3ca9a9c06ec08a826e8676a64718679b272c44d0 WatchSource:0}: Error finding container 9fbb5f3ffb8f74e77d89907e3ca9a9c06ec08a826e8676a64718679b272c44d0: Status 404 returned error can't find the container with id 9fbb5f3ffb8f74e77d89907e3ca9a9c06ec08a826e8676a64718679b272c44d0 Oct 30 17:58:39.390212 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:39.389960 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2g2gm" event={"ID":"24c78f02-e9ed-4344-b8fc-9a20c413548d","Type":"ContainerStarted","Data":"9fbb5f3ffb8f74e77d89907e3ca9a9c06ec08a826e8676a64718679b272c44d0"} Oct 30 17:58:39.391765 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:39.391427 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" event={"ID":"f0119471-101c-491b-8a67-ca0c104d2261","Type":"ContainerStarted","Data":"001950cb84bb555b299fed63c424b8c19c193532e4038eb40a7c2f6b322dd304"} Oct 30 17:58:40.122630 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.122597 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:40.125147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.125120 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1489b258-153a-47ac-8740-c4e9417b90c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d6r5\" (UID: \"1489b258-153a-47ac-8740-c4e9417b90c6\") " pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:40.194352 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.194296 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4d6r5" Oct 30 17:58:40.396923 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.396844 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" event={"ID":"f0119471-101c-491b-8a67-ca0c104d2261","Type":"ContainerStarted","Data":"a6ad02a9d84392cd2be053fbad414c9d531ccc569991aea6b038d53f1a89a220"} Oct 30 17:58:40.397311 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.397137 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" Oct 30 17:58:40.402118 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.402096 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" Oct 30 17:58:40.436266 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.436206 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vpfj4" podStartSLOduration=1.450689801 podStartE2EDuration="2.436193244s" totalImagesPullingTime="985.503443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:40.435725735 +0000 UTC m=+17.260120027" watchObservedRunningTime="2026-10-30 17:58:40.436193244 +0000 UTC m=+17.260587530" Oct 30 17:58:40.524633 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.524307 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4d6r5"] Oct 30 17:58:40.529860 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:40.529831 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1489b258_153a_47ac_8740_c4e9417b90c6.slice/crio-ae01741855d9f2c961046669f44e87a6fddba335d075ea7c5942f3e0a4089b9c WatchSource:0}: Error finding container ae01741855d9f2c961046669f44e87a6fddba335d075ea7c5942f3e0a4089b9c: Status 404 returned error can't find the container with id ae01741855d9f2c961046669f44e87a6fddba335d075ea7c5942f3e0a4089b9c Oct 30 17:58:40.915135 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:40.915101 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:41.008108 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:41.008077 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:41.402353 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:41.402312 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d6r5" event={"ID":"1489b258-153a-47ac-8740-c4e9417b90c6","Type":"ContainerStarted","Data":"40b75d2aac3b270227d9e09b2f9d8d076c58edf8de1c8c4d200949aeb3943053"} Oct 30 17:58:41.402779 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:41.402396 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d6r5" event={"ID":"1489b258-153a-47ac-8740-c4e9417b90c6","Type":"ContainerStarted","Data":"842e1b5ab6a8d05face0e6d4f00406d6f0b12497a62d25d9af83569bb8de5b57"} Oct 30 17:58:41.402779 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:41.402429 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d6r5" event={"ID":"1489b258-153a-47ac-8740-c4e9417b90c6","Type":"ContainerStarted","Data":"ae01741855d9f2c961046669f44e87a6fddba335d075ea7c5942f3e0a4089b9c"} Oct 30 17:58:43.412444 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:43.412402 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d6r5" event={"ID":"1489b258-153a-47ac-8740-c4e9417b90c6","Type":"ContainerStarted","Data":"8bf02417237e16b24b7bac950130816176346ec4813d2675634277809a1d8bf3"} Oct 30 17:58:43.608189 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:43.608150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:45.278778 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.278738 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:45.450725 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.450693 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:45.548699 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.548618 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:45.550999 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.550969 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-999c6"] Oct 30 17:58:45.554764 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.554742 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.570961 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.570150 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:58:45.571402 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.571363 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:58:45.581132 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.581078 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-999c6"] Oct 30 17:58:45.581132 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.581127 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:58:45.602237 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.602209 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v88tz\"" Oct 30 17:58:45.602425 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.602228 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 17:58:45.623265 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.623234 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zxgbz"] Oct 30 17:58:45.626948 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.626910 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.650922 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.650880 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rlt9d\"" Oct 30 17:58:45.651247 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.651221 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:58:45.651367 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.651330 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:58:45.664492 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.664461 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3b57703-781e-4309-a8de-db9cc9a63157-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.664608 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.664517 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89bq\" (UniqueName: \"kubernetes.io/projected/b3b57703-781e-4309-a8de-db9cc9a63157-kube-api-access-t89bq\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.664768 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.664699 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b57703-781e-4309-a8de-db9cc9a63157-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.664930 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.664784 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b57703-781e-4309-a8de-db9cc9a63157-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.674417 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.674390 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:58:45.674536 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.674447 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:58:45.677299 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.677276 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:45.765363 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765337 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89bq\" (UniqueName: \"kubernetes.io/projected/b3b57703-781e-4309-a8de-db9cc9a63157-kube-api-access-t89bq\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.765507 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765381 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-wtmp\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.765572 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765506 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b57703-781e-4309-a8de-db9cc9a63157-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.765617 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765594 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b57703-781e-4309-a8de-db9cc9a63157-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.765703 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765631 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-accelerators-collector-config\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.765703 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765666 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-root\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.765703 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765697 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnfd\" (UniqueName: \"kubernetes.io/projected/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-kube-api-access-rsnfd\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.765878 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765796 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-textfile\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.765878 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765842 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.765976 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765876 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3b57703-781e-4309-a8de-db9cc9a63157-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.766027 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.765972 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-tls\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.766027 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.766014 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-sys\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.766117 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.766043 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-metrics-client-ca\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.766421 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.766375 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b57703-781e-4309-a8de-db9cc9a63157-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.767063 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.767005 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:45.768401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.768375 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b57703-781e-4309-a8de-db9cc9a63157-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.768610 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.768594 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3b57703-781e-4309-a8de-db9cc9a63157-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.782362 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.782337 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89bq\" (UniqueName: \"kubernetes.io/projected/b3b57703-781e-4309-a8de-db9cc9a63157-kube-api-access-t89bq\") pod \"openshift-state-metrics-7bd7489957-999c6\" (UID: \"b3b57703-781e-4309-a8de-db9cc9a63157\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.867273 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867198 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-wtmp\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867273 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867234 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-accelerators-collector-config\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867273 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867260 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-root\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867321 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-root\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867354 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnfd\" (UniqueName: \"kubernetes.io/projected/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-kube-api-access-rsnfd\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867377 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-wtmp\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867393 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-textfile\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867419 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867459 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-tls\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867483 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-sys\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.867542 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.867502 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-metrics-client-ca\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.868051 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.868029 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-metrics-client-ca\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.868135 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.868102 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-sys\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.868225 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.868204 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-accelerators-collector-config\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.868399 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.868335 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-textfile\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.870200 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.870175 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-tls\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.870308 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.870243 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.882029 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.882005 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" Oct 30 17:58:45.884278 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.884260 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnfd\" (UniqueName: \"kubernetes.io/projected/f089c1a2-ca6d-486d-91b5-1d60a0d5d262-kube-api-access-rsnfd\") pod \"node-exporter-zxgbz\" (UID: \"f089c1a2-ca6d-486d-91b5-1d60a0d5d262\") " pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.945166 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.945138 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zxgbz" Oct 30 17:58:45.986808 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:45.986748 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:46.087319 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.087281 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:46.187863 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.187804 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:46.194324 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.194299 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-999c6"] Oct 30 17:58:46.200513 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:46.200486 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b57703_781e_4309_a8de_db9cc9a63157.slice/crio-bade6fa0205636f28e4ac1fcffb8be0d17d07d83614ad89e7401d740d5836d3e WatchSource:0}: Error finding container bade6fa0205636f28e4ac1fcffb8be0d17d07d83614ad89e7401d740d5836d3e: Status 404 returned error can't find the container with id bade6fa0205636f28e4ac1fcffb8be0d17d07d83614ad89e7401d740d5836d3e Oct 30 17:58:46.423994 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.423953 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zxgbz" event={"ID":"f089c1a2-ca6d-486d-91b5-1d60a0d5d262","Type":"ContainerStarted","Data":"5a8d913c4be6e29d62c8e1ce89a949222080b09aa9388ce118b1e24dce9c9746"} Oct 30 17:58:46.427573 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.427547 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" event={"ID":"b3b57703-781e-4309-a8de-db9cc9a63157","Type":"ContainerStarted","Data":"41b4d843138f22ab706971c426eedcb4fb9ca1e0ab2e7465ae5679cd4d14fe6c"} Oct 30 17:58:46.427703 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.427605 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" event={"ID":"b3b57703-781e-4309-a8de-db9cc9a63157","Type":"ContainerStarted","Data":"226a2846719e184bbf2133844472657c339c570219380f711b7f91dd59a3bce4"} Oct 30 17:58:46.427703 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.427636 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" event={"ID":"b3b57703-781e-4309-a8de-db9cc9a63157","Type":"ContainerStarted","Data":"bade6fa0205636f28e4ac1fcffb8be0d17d07d83614ad89e7401d740d5836d3e"} Oct 30 17:58:46.578706 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.578084 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rb89h" Oct 30 17:58:46.771412 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.771374 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:46.870489 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:46.870448 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:47.080160 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.080130 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:58:47.081761 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.081735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:47.085421 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.085398 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.117786 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117749 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:58:47.117969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117796 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:58:47.117969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117811 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:58:47.117969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117756 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:58:47.117969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117865 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:58:47.117969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117906 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:58:47.117969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117918 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:58:47.117969 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117958 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:58:47.118260 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.117989 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:58:47.118260 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.118038 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kv56z\"" Oct 30 17:58:47.118981 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.118958 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:58:47.178740 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178655 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-config-out\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.178740 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178709 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-web-config\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.178979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178804 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.178979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178861 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.178979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178901 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.178979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178939 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.178979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178957 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.178979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178977 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.179230 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.178996 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.179230 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.179012 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jh22\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-kube-api-access-5jh22\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.179230 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.179035 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-config-volume\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.179230 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.179051 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.179230 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.179066 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.279766 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.279730 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.279967 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.279774 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.279967 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.279800 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jh22\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-kube-api-access-5jh22\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.279967 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.279943 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-config-volume\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.279976 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280004 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280036 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-config-out\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280079 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-web-config\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280107 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280375 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280133 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280375 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280174 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280375 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280219 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280375 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280243 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280613 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280590 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280849 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280671 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.280849 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.280756 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.282916 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.282893 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-config-volume\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.283258 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.283224 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-config-out\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.283550 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.283529 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.283834 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.283794 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.284255 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.284211 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.284349 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.284261 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-web-config\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.284437 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.284420 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.285323 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.285299 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.286379 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.286353 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.314644 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.314612 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jh22\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-kube-api-access-5jh22\") pod \"alertmanager-main-0\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.407853 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.407765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:47.434283 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.434245 4582 generic.go:356] "Generic (PLEG): container finished" podID="f089c1a2-ca6d-486d-91b5-1d60a0d5d262" containerID="60908ed799f2d9e14ba8d21db988779c3677690a347616aa3e708d2657eb98ef" exitCode=0 Oct 30 17:58:47.434723 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.434328 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zxgbz" event={"ID":"f089c1a2-ca6d-486d-91b5-1d60a0d5d262","Type":"ContainerDied","Data":"60908ed799f2d9e14ba8d21db988779c3677690a347616aa3e708d2657eb98ef"} Oct 30 17:58:47.436868 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.436843 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" event={"ID":"b3b57703-781e-4309-a8de-db9cc9a63157","Type":"ContainerStarted","Data":"6508689fc73f05ccb50679445e4e76164c369a7baf8e0617c7042a2d0a4c684c"} Oct 30 17:58:47.456789 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.456758 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:58:47.525368 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.525325 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-999c6" podStartSLOduration=1.5410176789999999 podStartE2EDuration="2.52531333s" totalImagesPullingTime="984.295651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:47.524877956 +0000 UTC m=+24.349272249" watchObservedRunningTime="2026-10-30 17:58:47.52531333 +0000 UTC m=+24.349707619" Oct 30 17:58:47.552735 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.552702 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:47.788446 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.788378 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b56887585-wwl24"] Oct 30 17:58:47.813814 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.813784 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.816258 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.816228 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:58:47.821045 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:47.821018 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5835e54b_74f0_4ece_a77a_797505233920.slice/crio-fb08a3e9d8b39a85ebc01da17062f9137bf59bc2c2747a1087821606b2260a54 WatchSource:0}: Error finding container fb08a3e9d8b39a85ebc01da17062f9137bf59bc2c2747a1087821606b2260a54: Status 404 returned error can't find the container with id fb08a3e9d8b39a85ebc01da17062f9137bf59bc2c2747a1087821606b2260a54 Oct 30 17:58:47.828358 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.827245 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b56887585-wwl24"] Oct 30 17:58:47.828358 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.828303 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 17:58:47.828358 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.828353 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 17:58:47.828713 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.828423 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-21tfiqu679akr\"" Oct 30 17:58:47.828713 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.828487 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 17:58:47.828713 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.828534 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 17:58:47.828713 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.828569 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 17:58:47.828713 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.828628 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q5x6s\"" Oct 30 17:58:47.985859 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.985806 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.985859 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.985859 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f81431-fa44-44fa-8cee-2d7afb4323be-tmp\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.986113 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.985966 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-grpc-tls\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.986113 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.986060 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.986113 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.986100 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.986236 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.986211 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.986306 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.986291 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f81431-fa44-44fa-8cee-2d7afb4323be-metrics-client-ca\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.986416 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.986396 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwwl\" (UniqueName: \"kubernetes.io/projected/a2f81431-fa44-44fa-8cee-2d7afb4323be-kube-api-access-ggwwl\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:47.986538 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:47.986423 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-tls\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087241 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087151 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087241 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087225 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f81431-fa44-44fa-8cee-2d7afb4323be-metrics-client-ca\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087468 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087280 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwwl\" (UniqueName: \"kubernetes.io/projected/a2f81431-fa44-44fa-8cee-2d7afb4323be-kube-api-access-ggwwl\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087468 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087310 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-tls\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087468 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087339 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087468 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087364 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f81431-fa44-44fa-8cee-2d7afb4323be-tmp\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087468 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087386 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-grpc-tls\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087468 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087421 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.087468 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.087453 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.088154 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.088005 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f81431-fa44-44fa-8cee-2d7afb4323be-tmp\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.088154 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.088093 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f81431-fa44-44fa-8cee-2d7afb4323be-metrics-client-ca\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.090479 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.090440 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.090479 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.090457 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.090721 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.090686 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.090884 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.090864 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-grpc-tls\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.090940 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.090927 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.090982 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.090934 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a2f81431-fa44-44fa-8cee-2d7afb4323be-secret-thanos-querier-tls\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.105992 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.105970 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwwl\" (UniqueName: \"kubernetes.io/projected/a2f81431-fa44-44fa-8cee-2d7afb4323be-kube-api-access-ggwwl\") pod \"thanos-querier-7b56887585-wwl24\" (UID: \"a2f81431-fa44-44fa-8cee-2d7afb4323be\") " pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.205633 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.205599 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66c698cd5b-jpd4j"] Oct 30 17:58:48.208440 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.208414 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.226286 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.226255 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:58:48.226507 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.226491 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:58:48.226584 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.226515 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xxtjb\"" Oct 30 17:58:48.226617 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.226582 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:58:48.226648 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.226615 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:58:48.226648 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.226634 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:58:48.229643 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.229622 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c698cd5b-jpd4j"] Oct 30 17:58:48.231505 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.231482 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:48.389872 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.389833 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-service-ca\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.390052 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.389889 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-config\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.390052 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.389918 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-oauth-serving-cert\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.390052 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.389969 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jb7\" (UniqueName: \"kubernetes.io/projected/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-kube-api-access-q6jb7\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.390052 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.390001 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-tmp\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.390052 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.390049 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-serving-cert\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.390308 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.390086 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-oauth-config\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.444056 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.444018 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zxgbz" event={"ID":"f089c1a2-ca6d-486d-91b5-1d60a0d5d262","Type":"ContainerStarted","Data":"d1d33725710af4e13c15cb896b30a4c0f5724f26e85afbbb1cdc803cb341f53f"} Oct 30 17:58:48.444530 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.444082 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zxgbz" event={"ID":"f089c1a2-ca6d-486d-91b5-1d60a0d5d262","Type":"ContainerStarted","Data":"969154d566d32746e63f16c0281b5ced343b8f02095750b54eba1b8417c7a7f6"} Oct 30 17:58:48.445558 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.445528 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerStarted","Data":"fb08a3e9d8b39a85ebc01da17062f9137bf59bc2c2747a1087821606b2260a54"} Oct 30 17:58:48.491136 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491098 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-serving-cert\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.491321 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491157 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-oauth-config\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.491321 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491187 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-service-ca\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.491321 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491215 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-config\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.491321 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491240 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-oauth-serving-cert\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.491321 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491281 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jb7\" (UniqueName: \"kubernetes.io/projected/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-kube-api-access-q6jb7\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.491321 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491309 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-tmp\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.491731 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.491707 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-tmp\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.492841 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.492790 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-config\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.493023 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.492975 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-service-ca\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.493401 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.493362 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-oauth-serving-cert\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.495530 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.495507 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-oauth-config\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.496016 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.495994 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-serving-cert\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.497585 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.497535 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zxgbz" podStartSLOduration=2.472331008 podStartE2EDuration="3.497520239s" totalImagesPullingTime="951.750083ms" totalInitContainerRuntime="73.439148ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:48.49723641 +0000 UTC m=+25.321630701" watchObservedRunningTime="2026-10-30 17:58:48.497520239 +0000 UTC m=+25.321914525" Oct 30 17:58:48.528760 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.528724 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jb7\" (UniqueName: \"kubernetes.io/projected/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-kube-api-access-q6jb7\") pod \"console-66c698cd5b-jpd4j\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.531368 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.531342 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:48.558808 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.558632 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b56887585-wwl24"] Oct 30 17:58:48.566277 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:48.566250 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f81431_fa44_44fa_8cee_2d7afb4323be.slice/crio-abecec7ce20427bbd540df1068f9348cf4b36a710d71331da33baeaf624fabad WatchSource:0}: Error finding container abecec7ce20427bbd540df1068f9348cf4b36a710d71331da33baeaf624fabad: Status 404 returned error can't find the container with id abecec7ce20427bbd540df1068f9348cf4b36a710d71331da33baeaf624fabad Oct 30 17:58:48.801615 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.801518 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:48.841193 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.841163 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c698cd5b-jpd4j"] Oct 30 17:58:48.846265 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:48.846236 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4c9f32_ba9a_44d4_9adc_3d534f7111f5.slice/crio-0af8f7a18dd76808a7785b6b62588f463057debb3674fe7a7d3c657b5bc01ff6 WatchSource:0}: Error finding container 0af8f7a18dd76808a7785b6b62588f463057debb3674fe7a7d3c657b5bc01ff6: Status 404 returned error can't find the container with id 0af8f7a18dd76808a7785b6b62588f463057debb3674fe7a7d3c657b5bc01ff6 Oct 30 17:58:48.915202 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:48.915065 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:49.015436 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.015387 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:49.448194 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.448153 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:49.452672 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.452642 4582 generic.go:356] "Generic (PLEG): container finished" podID="5835e54b-74f0-4ece-a77a-797505233920" containerID="6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13" exitCode=0 Oct 30 17:58:49.452857 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.452734 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13"} Oct 30 17:58:49.454880 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.454853 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c698cd5b-jpd4j" event={"ID":"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5","Type":"ContainerStarted","Data":"0af8f7a18dd76808a7785b6b62588f463057debb3674fe7a7d3c657b5bc01ff6"} Oct 30 17:58:49.456299 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.456201 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" event={"ID":"a2f81431-fa44-44fa-8cee-2d7afb4323be","Type":"ContainerStarted","Data":"abecec7ce20427bbd540df1068f9348cf4b36a710d71331da33baeaf624fabad"} Oct 30 17:58:49.548343 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.548307 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:49.649373 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:49.649306 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:50.396988 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.396896 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:50.417916 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.417604 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-trwdq"] Oct 30 17:58:50.421208 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.421184 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.434265 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.434238 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 17:58:50.446391 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.446362 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-trwdq"] Oct 30 17:58:50.446534 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.446480 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t2ftw\"" Oct 30 17:58:50.493412 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.493374 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:50.614482 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.614435 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-trwdq\" (UID: \"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.614654 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.614487 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svncq\" (UniqueName: \"kubernetes.io/projected/5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4-kube-api-access-svncq\") pod \"monitoring-plugin-77f4f76594-trwdq\" (UID: \"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.632698 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.632612 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:50.715637 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.715548 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-trwdq\" (UID: \"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.715637 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.715606 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svncq\" (UniqueName: \"kubernetes.io/projected/5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4-kube-api-access-svncq\") pod \"monitoring-plugin-77f4f76594-trwdq\" (UID: \"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.720187 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.720159 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-trwdq\" (UID: \"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.735694 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.735660 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svncq\" (UniqueName: \"kubernetes.io/projected/5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4-kube-api-access-svncq\") pod \"monitoring-plugin-77f4f76594-trwdq\" (UID: \"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.737294 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.737270 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:50.907363 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.907325 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65db959845-drtds"] Oct 30 17:58:50.911687 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.911654 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.927736 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.927697 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-telemeter-client-tls\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.927909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.927775 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-metrics-client-ca\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.927909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.927833 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-secret-telemeter-client\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.927909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.927870 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-serving-certs-ca-bundle\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.928091 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.927925 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.928091 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.927988 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.928091 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.928028 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-federate-client-tls\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.928091 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.928066 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmc86\" (UniqueName: \"kubernetes.io/projected/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-kube-api-access-lmc86\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:50.938922 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.938026 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 17:58:50.938922 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.938169 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g2hp5\"" Oct 30 17:58:50.938922 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.938219 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 17:58:50.938922 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.938272 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 17:58:50.938922 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.938429 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 17:58:50.938922 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.938490 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 17:58:50.958473 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.958443 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 17:58:50.963290 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:50.963270 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65db959845-drtds"] Oct 30 17:58:51.028800 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028714 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-telemeter-client-tls\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.028800 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028772 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-metrics-client-ca\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.029031 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028802 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-secret-telemeter-client\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.029031 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028847 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-serving-certs-ca-bundle\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.029031 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028881 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.029031 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028926 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.029031 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028955 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-federate-client-tls\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.029031 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.028998 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmc86\" (UniqueName: \"kubernetes.io/projected/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-kube-api-access-lmc86\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.030372 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.030348 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-serving-certs-ca-bundle\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.030971 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.030948 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-metrics-client-ca\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.032253 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.032195 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.047659 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.047612 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-secret-telemeter-client\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.049616 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.049588 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.049840 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.049804 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-telemeter-client-tls\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.051382 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.051333 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-federate-client-tls\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.052459 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.052440 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmc86\" (UniqueName: \"kubernetes.io/projected/ef2f4767-ef82-4e82-bba6-3857a52f3a3d-kube-api-access-lmc86\") pod \"telemeter-client-65db959845-drtds\" (UID: \"ef2f4767-ef82-4e82-bba6-3857a52f3a3d\") " pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.067672 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.067590 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-trwdq"] Oct 30 17:58:51.073343 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:51.073309 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1f58b2_45e1_420b_be68_f6ff9e2eb0c4.slice/crio-e97aff6602645812e240e18c6a09ca0e8008fd78af0b482570e3340e69643bb1 WatchSource:0}: Error finding container e97aff6602645812e240e18c6a09ca0e8008fd78af0b482570e3340e69643bb1: Status 404 returned error can't find the container with id e97aff6602645812e240e18c6a09ca0e8008fd78af0b482570e3340e69643bb1 Oct 30 17:58:51.247586 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.247474 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65db959845-drtds" Oct 30 17:58:51.477627 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.477594 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" event={"ID":"a2f81431-fa44-44fa-8cee-2d7afb4323be","Type":"ContainerStarted","Data":"ac90e5d2787e84842bb65de792a397d3b46ab55ae8515842f14ca1d9304720a2"} Oct 30 17:58:51.477801 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.477672 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" event={"ID":"a2f81431-fa44-44fa-8cee-2d7afb4323be","Type":"ContainerStarted","Data":"1a8be165d4e5f71674a9661266ac0fb94c167df30d84cf918cb9b751caa29584"} Oct 30 17:58:51.477801 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.477715 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" event={"ID":"a2f81431-fa44-44fa-8cee-2d7afb4323be","Type":"ContainerStarted","Data":"b0d6eb5b9e645c60b17abb5a0711f8128880dca16ff96bdf83faa5a51d4711a5"} Oct 30 17:58:51.479243 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.479217 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" event={"ID":"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4","Type":"ContainerStarted","Data":"e97aff6602645812e240e18c6a09ca0e8008fd78af0b482570e3340e69643bb1"} Oct 30 17:58:51.594241 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:51.594213 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65db959845-drtds"] Oct 30 17:58:51.599284 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:58:51.599252 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2f4767_ef82_4e82_bba6_3857a52f3a3d.slice/crio-cbbfeb09f02bf0a83e057f66b5018c6104a5dfc3bf9c3ae8c9a0f6a9288682bf WatchSource:0}: Error finding container cbbfeb09f02bf0a83e057f66b5018c6104a5dfc3bf9c3ae8c9a0f6a9288682bf: Status 404 returned error can't find the container with id cbbfeb09f02bf0a83e057f66b5018c6104a5dfc3bf9c3ae8c9a0f6a9288682bf Oct 30 17:58:52.490634 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.490374 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerStarted","Data":"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9"} Oct 30 17:58:52.490634 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.490457 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerStarted","Data":"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48"} Oct 30 17:58:52.490634 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.490501 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerStarted","Data":"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6"} Oct 30 17:58:52.490634 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.490540 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerStarted","Data":"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447"} Oct 30 17:58:52.490634 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.490579 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerStarted","Data":"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8"} Oct 30 17:58:52.494701 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.494671 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" event={"ID":"a2f81431-fa44-44fa-8cee-2d7afb4323be","Type":"ContainerStarted","Data":"03901cdfb10405d85db038182ac9a23d2b9517860b58cdc91f2c89d1c6522f46"} Oct 30 17:58:52.496059 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.496036 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65db959845-drtds" event={"ID":"ef2f4767-ef82-4e82-bba6-3857a52f3a3d","Type":"ContainerStarted","Data":"cbbfeb09f02bf0a83e057f66b5018c6104a5dfc3bf9c3ae8c9a0f6a9288682bf"} Oct 30 17:58:52.629094 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.628018 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:58:52.635448 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.635427 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.645371 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.645345 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.645815 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.645787 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.645921 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.645850 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.645921 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.645892 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646044 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.645948 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646044 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.645986 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646052 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646092 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646147 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646124 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646147 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkmj\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-kube-api-access-rjkmj\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646183 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646206 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646231 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646257 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646596 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646324 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-config\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646596 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646352 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646596 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646376 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646596 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646414 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646596 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646439 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.646596 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.646465 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.662128 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.662039 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 17:58:52.662388 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.662358 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 17:58:52.662629 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.662479 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:58:52.662973 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.662882 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 17:58:52.683869 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.683531 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 17:58:52.683869 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.683587 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 17:58:52.683869 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.683693 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 17:58:52.683869 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.683761 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 17:58:52.683869 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.683851 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 17:58:52.684316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684016 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8go9m9pn67jdm\"" Oct 30 17:58:52.684316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684145 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 17:58:52.684316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684264 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x2p2x\"" Oct 30 17:58:52.684453 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684329 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 17:58:52.684453 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684426 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 17:58:52.684570 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684552 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 17:58:52.684682 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684667 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 17:58:52.684752 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.684739 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 17:58:52.690910 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.688291 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:58:52.729464 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.729432 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:52.747939 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.747910 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748067 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.747956 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748067 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.747989 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748067 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748028 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748067 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748054 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748211 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748077 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748211 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748141 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748211 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748174 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748211 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748203 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748325 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748227 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkmj\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-kube-api-access-rjkmj\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748325 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748262 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748325 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748286 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748325 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748312 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748427 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748338 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748427 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748364 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-config\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748427 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748393 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748528 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748452 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748528 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748491 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748587 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748515 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.748587 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.748550 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.749026 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.749004 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.750708 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.750681 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.751661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.751016 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.751661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.751260 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.751661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.751423 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.752747 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.752722 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.753400 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.753376 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.754602 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.754142 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.754602 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.754188 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.754602 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.754230 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.754602 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.754269 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.754602 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.754562 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-config\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.755716 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.755665 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.758243 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.758221 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.758979 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.758919 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.759652 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.759356 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.759652 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.759481 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.759652 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.759600 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.761704 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.761670 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.767582 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.767560 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkmj\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-kube-api-access-rjkmj\") pod \"prometheus-k8s-0\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:52.998113 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:52.997876 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:58:53.408715 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:53.408677 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:58:53.505327 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:53.505288 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerStarted","Data":"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb"} Oct 30 17:58:53.509624 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:53.509595 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" event={"ID":"a2f81431-fa44-44fa-8cee-2d7afb4323be","Type":"ContainerStarted","Data":"ce9cd73ab199374d7f44f2db3843fb28433168c8f60a5c70159c12a64176b3ec"} Oct 30 17:58:53.509742 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:53.509665 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" event={"ID":"a2f81431-fa44-44fa-8cee-2d7afb4323be","Type":"ContainerStarted","Data":"def8683f62a49565a2ba4313c201048786ef337b9ebbcf42abd6c4f2723b4c6d"} Oct 30 17:58:53.509742 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:53.509718 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:53.552243 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:53.551777 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.797025934 podStartE2EDuration="6.551762434s" totalImagesPullingTime="3.740027146s" totalInitContainerRuntime="14.709354ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:53.54954492 +0000 UTC m=+30.373939223" watchObservedRunningTime="2026-10-30 17:58:53.551762434 +0000 UTC m=+30.376156725" Oct 30 17:58:53.590847 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:53.590756 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" podStartSLOduration=3.024144412 podStartE2EDuration="6.590733892s" totalImagesPullingTime="3.56658948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:53.589630147 +0000 UTC m=+30.414024479" watchObservedRunningTime="2026-10-30 17:58:53.590733892 +0000 UTC m=+30.415128186" Oct 30 17:58:54.516424 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.516390 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" event={"ID":"5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4","Type":"ContainerStarted","Data":"6a0cca16d7267627a6339641ff4940ea61e84ea5b7c15038f5539fa8637d5b36"} Oct 30 17:58:54.516841 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.516533 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:54.518657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.518536 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a10818-9ffe-499d-b728-8b1591912823" containerID="d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989" exitCode=0 Oct 30 17:58:54.518657 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.518614 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989"} Oct 30 17:58:54.518802 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.518688 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerStarted","Data":"eced99d37b1420b29a0dfe154eebf8d629b3f53db226cba08cdddde14f9267e9"} Oct 30 17:58:54.521895 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.521873 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" Oct 30 17:58:54.522242 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.522210 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c698cd5b-jpd4j" event={"ID":"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5","Type":"ContainerStarted","Data":"27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941"} Oct 30 17:58:54.525809 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.525788 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65db959845-drtds" event={"ID":"ef2f4767-ef82-4e82-bba6-3857a52f3a3d","Type":"ContainerStarted","Data":"f394db7f84a0867f0446103d3409af7e61dc46271aa2498085b369f2f637bfaa"} Oct 30 17:58:54.525937 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.525920 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65db959845-drtds" event={"ID":"ef2f4767-ef82-4e82-bba6-3857a52f3a3d","Type":"ContainerStarted","Data":"a46ae0b1234559e5890cd8d636084da8875bba63b5b6a6792b26a532e25351b9"} Oct 30 17:58:54.525986 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.525956 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65db959845-drtds" event={"ID":"ef2f4767-ef82-4e82-bba6-3857a52f3a3d","Type":"ContainerStarted","Data":"1c98725be44561b87cb62e13335863f85b04773dae5fe8a0ecca52f486b53ca3"} Oct 30 17:58:54.554129 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.554080 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-trwdq" podStartSLOduration=1.810743001 podStartE2EDuration="4.554062379s" totalImagesPullingTime="2.743319378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:54.553570225 +0000 UTC m=+31.377964516" watchObservedRunningTime="2026-10-30 17:58:54.554062379 +0000 UTC m=+31.378456668" Oct 30 17:58:54.627810 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.627780 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:54.742055 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.741945 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66c698cd5b-jpd4j" podStartSLOduration=1.767962252 podStartE2EDuration="6.741927125s" totalImagesPullingTime="4.973964873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:54.741656413 +0000 UTC m=+31.566050706" watchObservedRunningTime="2026-10-30 17:58:54.741927125 +0000 UTC m=+31.566321418" Oct 30 17:58:54.827632 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:54.827591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:55.588725 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:55.588658 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65db959845-drtds" podStartSLOduration=2.986276651 podStartE2EDuration="5.588644134s" totalImagesPullingTime="2.602367483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:55.578495465 +0000 UTC m=+32.402889759" watchObservedRunningTime="2026-10-30 17:58:55.588644134 +0000 UTC m=+32.413038425" Oct 30 17:58:57.540106 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:57.540066 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2g2gm" event={"ID":"24c78f02-e9ed-4344-b8fc-9a20c413548d","Type":"ContainerStarted","Data":"4904aabf1733c852e187acaa7912fdc3b688e7ee6b5b170491c012673e12e01f"} Oct 30 17:58:57.540562 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:57.540296 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:57.542056 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:57.542027 4582 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-2g2gm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused" start-of-body= Oct 30 17:58:57.542200 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:57.542091 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-2g2gm" podUID="24c78f02-e9ed-4344-b8fc-9a20c413548d" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused" Oct 30 17:58:57.576085 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:57.575999 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-2g2gm" podStartSLOduration=1.13754214 podStartE2EDuration="19.575985285s" totalImagesPullingTime="18.438443145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:57.575856465 +0000 UTC m=+34.400250785" watchObservedRunningTime="2026-10-30 17:58:57.575985285 +0000 UTC m=+34.400379576" Oct 30 17:58:58.413166 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:58.413130 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c5qcf" Oct 30 17:58:58.532579 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:58.532549 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:58.532763 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:58.532618 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:58.538397 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:58.538371 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:58.549022 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:58.548998 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:58:58.555628 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:58.555601 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-2g2gm" Oct 30 17:58:59.544892 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:59.542582 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b56887585-wwl24" Oct 30 17:58:59.554553 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:59.554528 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerStarted","Data":"d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7"} Oct 30 17:58:59.554982 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:59.554596 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerStarted","Data":"0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1"} Oct 30 17:58:59.554982 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:59.554643 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerStarted","Data":"595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21"} Oct 30 17:58:59.554982 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:58:59.554687 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerStarted","Data":"ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f"} Oct 30 17:59:00.174423 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.174389 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:00.307263 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.307224 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:00.439608 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.439521 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:00.562681 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.562640 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerStarted","Data":"d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede"} Oct 30 17:59:00.563138 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.562728 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerStarted","Data":"08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996"} Oct 30 17:59:00.576191 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.576143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:00.629307 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.629252 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.123798702 podStartE2EDuration="8.629235249s" totalImagesPullingTime="4.470700795s" totalInitContainerRuntime="34.735752ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:00.626227467 +0000 UTC m=+37.450621760" watchObservedRunningTime="2026-10-30 17:59:00.629235249 +0000 UTC m=+37.453629542" Oct 30 17:59:00.710761 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.710688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:00.931618 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:00.931578 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:02.745912 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:02.745868 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:02.999094 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:02.999004 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:59:03.800325 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:03.800265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:03.904292 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:03.904261 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:06.223795 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:06.223762 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 17:59:10.165091 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:10.165046 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:10.245455 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:10.245419 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:11.367504 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:11.367467 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c698cd5b-jpd4j"] Oct 30 17:59:11.430983 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:11.430950 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:11.539632 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:11.539601 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:16.945655 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:16.945620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:17.047615 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:17.047585 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:18.805488 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:18.805452 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:18.905874 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:18.905842 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:28.016126 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.016088 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hn6rb-55v8r"] Oct 30 17:59:28.016693 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.016355 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" podUID="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" containerName="insights-gathering" containerID="cri-o://b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698" gracePeriod=30 Oct 30 17:59:28.347368 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.347343 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:59:28.475764 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.475726 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\") pod \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " Oct 30 17:59:28.475936 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.475813 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\" (UniqueName: \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\") pod \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " Oct 30 17:59:28.475936 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.475872 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\") pod \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " Oct 30 17:59:28.476015 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.475937 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\") pod \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\" (UID: \"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022\") " Oct 30 17:59:28.476225 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.476199 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle" pod "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" (UID: "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:28.476264 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.476225 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle" pod "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" (UID: "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:28.476316 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.476290 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path" pod "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" (UID: "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:28.478116 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.478093 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9" pod "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" (UID: "a02d9ec7-7d1e-4d1a-a24e-1bf17f279022"). InnerVolumeSpecName "kube-api-access-4ljf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:28.576691 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.576625 4582 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-archives-path\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:28.576691 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.576650 4582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-trusted-ca-bundle\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:28.576691 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.576660 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljf9\" (UniqueName: \"kubernetes.io/projected/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-kube-api-access-4ljf9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:28.576691 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.576670 4582 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022-service-ca-bundle\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:28.670964 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.670931 4582 generic.go:356] "Generic (PLEG): container finished" podID="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" containerID="b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698" exitCode=2 Oct 30 17:59:28.671101 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.670990 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" Oct 30 17:59:28.671101 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.671017 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" event={"ID":"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022","Type":"ContainerDied","Data":"b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698"} Oct 30 17:59:28.671101 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.671066 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hn6rb-55v8r" event={"ID":"a02d9ec7-7d1e-4d1a-a24e-1bf17f279022","Type":"ContainerDied","Data":"347b895d1f06e3ce09d22ae9fb1894eaed119371e8e5a329f667f977a2c1ac1f"} Oct 30 17:59:28.671101 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.671089 4582 scope.go:122] "RemoveContainer" containerID="b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698" Oct 30 17:59:28.684705 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.684687 4582 scope.go:122] "RemoveContainer" containerID="b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698" Oct 30 17:59:28.684993 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:28.684974 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698\": container with ID starting with b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698 not found: ID does not exist" containerID="b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698" Oct 30 17:59:28.685043 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.685002 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698"} err="failed to get container status \"b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698\": rpc error: code = NotFound desc = could not find container \"b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698\": container with ID starting with b173bfe1bc25ab8a5c3b658e88e47992383a2a28f8825cad0c2af6ea917a7698 not found: ID does not exist" Oct 30 17:59:28.713048 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.713026 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hn6rb-55v8r"] Oct 30 17:59:28.718572 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:28.718551 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hn6rb-55v8r"] Oct 30 17:59:29.297733 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:29.297700 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" path="/var/lib/kubelet/pods/a02d9ec7-7d1e-4d1a-a24e-1bf17f279022/volumes" Oct 30 17:59:33.524072 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:33.524035 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:33.603115 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:33.603083 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:33.793672 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:33.793561 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:33.890968 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:33.890917 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:36.223449 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:36.223414 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 17:59:36.387470 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.387434 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66c698cd5b-jpd4j" podUID="7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" containerName="console" containerID="cri-o://27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941" gracePeriod=15 Oct 30 17:59:36.659531 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.659507 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:59:36.703142 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.703108 4582 generic.go:356] "Generic (PLEG): container finished" podID="7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" containerID="27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941" exitCode=0 Oct 30 17:59:36.703278 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.703153 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c698cd5b-jpd4j" Oct 30 17:59:36.703278 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.703192 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c698cd5b-jpd4j" event={"ID":"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5","Type":"ContainerDied","Data":"27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941"} Oct 30 17:59:36.703278 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.703228 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c698cd5b-jpd4j" event={"ID":"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5","Type":"ContainerDied","Data":"0af8f7a18dd76808a7785b6b62588f463057debb3674fe7a7d3c657b5bc01ff6"} Oct 30 17:59:36.703278 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.703250 4582 scope.go:122] "RemoveContainer" containerID="27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941" Oct 30 17:59:36.711157 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.711137 4582 scope.go:122] "RemoveContainer" containerID="27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941" Oct 30 17:59:36.711414 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:36.711391 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941\": container with ID starting with 27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941 not found: ID does not exist" containerID="27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941" Oct 30 17:59:36.711498 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.711420 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941"} err="failed to get container status \"27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941\": rpc error: code = NotFound desc = could not find container \"27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941\": container with ID starting with 27a943f5a81ed25ffca3641e685d7cb90c32ee15f5c0f8697af3f113473d3941 not found: ID does not exist" Oct 30 17:59:36.741772 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.741749 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-oauth-serving-cert\") pod \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " Oct 30 17:59:36.741875 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.741790 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-oauth-config\") pod \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " Oct 30 17:59:36.741875 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.741814 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-service-ca\") pod \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " Oct 30 17:59:36.741875 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.741858 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-config\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-config\") pod \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " Oct 30 17:59:36.742023 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.741875 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-serving-cert\") pod \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " Oct 30 17:59:36.742023 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.741899 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-tmp\") pod \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " Oct 30 17:59:36.742023 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.741938 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-kube-api-access-q6jb7\" (UniqueName: \"kubernetes.io/projected/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-kube-api-access-q6jb7\") pod \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\" (UID: \"7f4c9f32-ba9a-44d4-9adc-3d534f7111f5\") " Oct 30 17:59:36.742260 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.742201 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-tmp" pod "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" (UID: "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:36.742322 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.742263 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-service-ca" pod "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" (UID: "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:36.742372 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.742309 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-config" pod "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" (UID: "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:36.742372 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.742361 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-oauth-serving-cert" pod "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" (UID: "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:36.744086 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.744065 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-kube-api-access-q6jb7" pod "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" (UID: "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5"). InnerVolumeSpecName "kube-api-access-q6jb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:36.744256 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.744239 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-oauth-config" pod "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" (UID: "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:36.744323 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.744264 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-serving-cert" pod "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" (UID: "7f4c9f32-ba9a-44d4-9adc-3d534f7111f5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:36.843422 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.843358 4582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-oauth-config\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:36.843422 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.843383 4582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:36.843422 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.843393 4582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-config\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:36.843422 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.843402 4582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-console-serving-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:36.843422 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.843410 4582 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-tmp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:36.843422 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.843420 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jb7\" (UniqueName: \"kubernetes.io/projected/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-kube-api-access-q6jb7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:36.843713 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:36.843429 4582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5-oauth-serving-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:37.042850 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:37.042808 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c698cd5b-jpd4j"] Oct 30 17:59:37.052101 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:37.052078 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66c698cd5b-jpd4j"] Oct 30 17:59:37.297008 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:37.296973 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" path="/var/lib/kubelet/pods/7f4c9f32-ba9a-44d4-9adc-3d534f7111f5/volumes" Oct 30 17:59:39.302410 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:39.302377 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:44.212301 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:44.212261 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:59:44.256902 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:44.256877 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:59:44.748984 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:44.748959 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:59:52.413233 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:52.413193 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:52.511992 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:52.511964 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:57.485894 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.485862 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:59:57.486395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.486119 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="alertmanager" containerID="cri-o://103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8" gracePeriod=120 Oct 30 17:59:57.486395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.486202 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-metric" containerID="cri-o://68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9" gracePeriod=120 Oct 30 17:59:57.486395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.486219 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="prom-label-proxy" containerID="cri-o://ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb" gracePeriod=120 Oct 30 17:59:57.486395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.486261 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-web" containerID="cri-o://ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6" gracePeriod=120 Oct 30 17:59:57.486395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.486304 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy" containerID="cri-o://570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48" gracePeriod=120 Oct 30 17:59:57.486395 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.486273 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="config-reloader" containerID="cri-o://9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447" gracePeriod=120 Oct 30 17:59:57.789611 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789531 4582 generic.go:356] "Generic (PLEG): container finished" podID="5835e54b-74f0-4ece-a77a-797505233920" containerID="ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb" exitCode=0 Oct 30 17:59:57.789611 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789557 4582 generic.go:356] "Generic (PLEG): container finished" podID="5835e54b-74f0-4ece-a77a-797505233920" containerID="570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48" exitCode=0 Oct 30 17:59:57.789611 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789564 4582 generic.go:356] "Generic (PLEG): container finished" podID="5835e54b-74f0-4ece-a77a-797505233920" containerID="9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447" exitCode=0 Oct 30 17:59:57.789611 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789571 4582 generic.go:356] "Generic (PLEG): container finished" podID="5835e54b-74f0-4ece-a77a-797505233920" containerID="103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8" exitCode=0 Oct 30 17:59:57.789909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789609 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb"} Oct 30 17:59:57.789909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789684 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48"} Oct 30 17:59:57.789909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789724 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447"} Oct 30 17:59:57.789909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:57.789761 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8"} Oct 30 17:59:58.727045 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.727018 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:58.796351 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.796322 4582 generic.go:356] "Generic (PLEG): container finished" podID="5835e54b-74f0-4ece-a77a-797505233920" containerID="68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9" exitCode=0 Oct 30 17:59:58.796351 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.796343 4582 generic.go:356] "Generic (PLEG): container finished" podID="5835e54b-74f0-4ece-a77a-797505233920" containerID="ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6" exitCode=0 Oct 30 17:59:58.796518 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.796407 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:58.796518 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.796405 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9"} Oct 30 17:59:58.796588 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.796529 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6"} Oct 30 17:59:58.796588 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.796556 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5835e54b-74f0-4ece-a77a-797505233920","Type":"ContainerDied","Data":"fb08a3e9d8b39a85ebc01da17062f9137bf59bc2c2747a1087821606b2260a54"} Oct 30 17:59:58.796588 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.796585 4582 scope.go:122] "RemoveContainer" containerID="ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb" Oct 30 17:59:58.806018 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.805995 4582 scope.go:122] "RemoveContainer" containerID="68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9" Oct 30 17:59:58.813522 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.813506 4582 scope.go:122] "RemoveContainer" containerID="570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48" Oct 30 17:59:58.820466 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.820439 4582 scope.go:122] "RemoveContainer" containerID="ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6" Oct 30 17:59:58.827750 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.827727 4582 scope.go:122] "RemoveContainer" containerID="9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447" Oct 30 17:59:58.835532 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.835516 4582 scope.go:122] "RemoveContainer" containerID="103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8" Oct 30 17:59:58.842679 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.842656 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-tls-assets\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-tls-assets\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.842757 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.842695 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-trusted-ca-bundle\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.842757 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.842721 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.842878 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.842852 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-main-db\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.842950 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.842933 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-web\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.843003 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.842980 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-kube-api-access-5jh22\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-kube-api-access-5jh22\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.843063 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843012 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-web-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-web-config\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.843063 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843049 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-cluster-tls-config\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.843165 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843094 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-config-volume\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-config-volume\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.843165 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843095 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-trusted-ca-bundle" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:58.843165 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843130 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-config-out\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.843311 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843203 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-main-tls\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.843311 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843249 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.845884 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.844383 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-main-db" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:58.845884 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.843284 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-metrics-client-ca\") pod \"5835e54b-74f0-4ece-a77a-797505233920\" (UID: \"5835e54b-74f0-4ece-a77a-797505233920\") " Oct 30 17:59:58.846648 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.846483 4582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.846648 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.846515 4582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-alertmanager-main-db\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.847417 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.847390 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-kube-api-access-5jh22" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "kube-api-access-5jh22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:58.847516 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.847464 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-tls-assets" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:58.847582 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.847552 4582 scope.go:122] "RemoveContainer" containerID="6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13" Oct 30 17:59:58.848058 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.848017 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-metrics-client-ca" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:58.848213 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.848162 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:58.848774 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.848696 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-main-tls" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:58.855974 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.855946 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-config-out" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:58.856078 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.855981 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-metric" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:58.856078 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.856055 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-config-volume" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:58.857564 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.857524 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-web" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:58.858190 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.858168 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-cluster-tls-config" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:58.863127 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.863106 4582 scope.go:122] "RemoveContainer" containerID="ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb" Oct 30 17:59:58.863427 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:58.863408 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb\": container with ID starting with ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb not found: ID does not exist" containerID="ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb" Oct 30 17:59:58.863524 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.863434 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb"} err="failed to get container status \"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb\": rpc error: code = NotFound desc = could not find container \"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb\": container with ID starting with ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb not found: ID does not exist" Oct 30 17:59:58.863524 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.863453 4582 scope.go:122] "RemoveContainer" containerID="68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9" Oct 30 17:59:58.863703 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:58.863684 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9\": container with ID starting with 68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9 not found: ID does not exist" containerID="68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9" Oct 30 17:59:58.863760 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.863707 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9"} err="failed to get container status \"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9\": rpc error: code = NotFound desc = could not find container \"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9\": container with ID starting with 68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9 not found: ID does not exist" Oct 30 17:59:58.863760 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.863720 4582 scope.go:122] "RemoveContainer" containerID="570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48" Oct 30 17:59:58.864053 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:58.864035 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48\": container with ID starting with 570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48 not found: ID does not exist" containerID="570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48" Oct 30 17:59:58.864116 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864058 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48"} err="failed to get container status \"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48\": rpc error: code = NotFound desc = could not find container \"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48\": container with ID starting with 570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48 not found: ID does not exist" Oct 30 17:59:58.864116 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864074 4582 scope.go:122] "RemoveContainer" containerID="ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6" Oct 30 17:59:58.864365 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:58.864339 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6\": container with ID starting with ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6 not found: ID does not exist" containerID="ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6" Oct 30 17:59:58.864436 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864372 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6"} err="failed to get container status \"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6\": rpc error: code = NotFound desc = could not find container \"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6\": container with ID starting with ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6 not found: ID does not exist" Oct 30 17:59:58.864436 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864386 4582 scope.go:122] "RemoveContainer" containerID="9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447" Oct 30 17:59:58.864620 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:58.864604 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447\": container with ID starting with 9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447 not found: ID does not exist" containerID="9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447" Oct 30 17:59:58.864677 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864621 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447"} err="failed to get container status \"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447\": rpc error: code = NotFound desc = could not find container \"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447\": container with ID starting with 9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447 not found: ID does not exist" Oct 30 17:59:58.864677 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864636 4582 scope.go:122] "RemoveContainer" containerID="103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8" Oct 30 17:59:58.864895 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:58.864873 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8\": container with ID starting with 103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8 not found: ID does not exist" containerID="103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8" Oct 30 17:59:58.864967 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864904 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8"} err="failed to get container status \"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8\": rpc error: code = NotFound desc = could not find container \"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8\": container with ID starting with 103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8 not found: ID does not exist" Oct 30 17:59:58.864967 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.864924 4582 scope.go:122] "RemoveContainer" containerID="6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13" Oct 30 17:59:58.865176 ip-10-0-133-124 kubenswrapper[4582]: E1030 17:59:58.865158 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13\": container with ID starting with 6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13 not found: ID does not exist" containerID="6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13" Oct 30 17:59:58.865241 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865179 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13"} err="failed to get container status \"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13\": rpc error: code = NotFound desc = could not find container \"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13\": container with ID starting with 6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13 not found: ID does not exist" Oct 30 17:59:58.865241 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865191 4582 scope.go:122] "RemoveContainer" containerID="ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb" Oct 30 17:59:58.865431 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865403 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb"} err="failed to get container status \"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb\": rpc error: code = NotFound desc = could not find container \"ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb\": container with ID starting with ae051adf0ae07bff88b72f7952f0a35e6014f78760399d9b219461df8c3c87cb not found: ID does not exist" Oct 30 17:59:58.865431 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865430 4582 scope.go:122] "RemoveContainer" containerID="68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9" Oct 30 17:59:58.865661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865644 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9"} err="failed to get container status \"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9\": rpc error: code = NotFound desc = could not find container \"68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9\": container with ID starting with 68c7867a48efe0f8eaddd71301788d3f39553a74c8f0a58f65e33e6c3a8d63e9 not found: ID does not exist" Oct 30 17:59:58.865661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865651 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-web-config" pod "5835e54b-74f0-4ece-a77a-797505233920" (UID: "5835e54b-74f0-4ece-a77a-797505233920"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:58.865661 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865661 4582 scope.go:122] "RemoveContainer" containerID="570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48" Oct 30 17:59:58.865904 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865881 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48"} err="failed to get container status \"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48\": rpc error: code = NotFound desc = could not find container \"570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48\": container with ID starting with 570de27b5b21213bc9b36b1af9cfdc1690c58366d0212a5d10e798162d0f1d48 not found: ID does not exist" Oct 30 17:59:58.865964 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.865905 4582 scope.go:122] "RemoveContainer" containerID="ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6" Oct 30 17:59:58.866159 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.866142 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6"} err="failed to get container status \"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6\": rpc error: code = NotFound desc = could not find container \"ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6\": container with ID starting with ed3c4c34e635aa456379da400d1a62dd272d10aa3462ca540c1e316dcfb40fd6 not found: ID does not exist" Oct 30 17:59:58.866219 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.866160 4582 scope.go:122] "RemoveContainer" containerID="9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447" Oct 30 17:59:58.866384 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.866365 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447"} err="failed to get container status \"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447\": rpc error: code = NotFound desc = could not find container \"9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447\": container with ID starting with 9c3317c1045c5c3b9da0cb7e26a08e99d57b10fd9f7abfac3ae56144f1052447 not found: ID does not exist" Oct 30 17:59:58.866384 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.866382 4582 scope.go:122] "RemoveContainer" containerID="103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8" Oct 30 17:59:58.866589 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.866567 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8"} err="failed to get container status \"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8\": rpc error: code = NotFound desc = could not find container \"103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8\": container with ID starting with 103f8b555caea6b6d7115b0927ef9addb555adb4563fd5bfe61c49fc6b5ae1f8 not found: ID does not exist" Oct 30 17:59:58.866650 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.866590 4582 scope.go:122] "RemoveContainer" containerID="6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13" Oct 30 17:59:58.866785 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.866769 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13"} err="failed to get container status \"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13\": rpc error: code = NotFound desc = could not find container \"6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13\": container with ID starting with 6fe3178f07b6e0e498889fbd4800734ae38752fdc43e5669ab6a0f36acb97c13 not found: ID does not exist" Oct 30 17:59:58.947288 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947247 4582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-main-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947288 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947281 4582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947288 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947292 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5835e54b-74f0-4ece-a77a-797505233920-metrics-client-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947288 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947301 4582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-tls-assets\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947513 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947310 4582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947513 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947319 4582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947513 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947328 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jh22\" (UniqueName: \"kubernetes.io/projected/5835e54b-74f0-4ece-a77a-797505233920-kube-api-access-5jh22\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947513 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947350 4582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-web-config\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947513 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947359 4582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-cluster-tls-config\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947513 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947367 4582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5835e54b-74f0-4ece-a77a-797505233920-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:58.947513 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:58.947376 4582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5835e54b-74f0-4ece-a77a-797505233920-config-out\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 17:59:59.139402 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.139367 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:59:59.149157 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.149134 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:59:59.170640 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.170613 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:59:59.171072 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171054 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="prom-label-proxy" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171072 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="prom-label-proxy" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171090 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" containerName="console" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171099 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" containerName="console" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171111 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-web" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171117 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-web" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171140 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="alertmanager" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171148 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="alertmanager" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171158 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy" Oct 30 17:59:59.171162 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171165 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171174 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-metric" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171182 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-metric" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171190 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="init-config-reloader" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171197 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="init-config-reloader" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171208 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="config-reloader" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171215 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="config-reloader" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171224 4582 cpu_manager.go:414] "containerMap: removing container" podUID="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" containerName="insights-gathering" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171231 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" containerName="insights-gathering" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171303 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="alertmanager" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171311 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171317 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-metric" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171323 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="config-reloader" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171329 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02d9ec7-7d1e-4d1a-a24e-1bf17f279022" containerName="insights-gathering" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171335 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="prom-label-proxy" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171341 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4c9f32-ba9a-44d4-9adc-3d534f7111f5" containerName="console" Oct 30 17:59:59.171561 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.171347 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835e54b-74f0-4ece-a77a-797505233920" containerName="kube-rbac-proxy-web" Oct 30 17:59:59.175454 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.175439 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.188062 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188042 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:59:59.188153 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188060 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:59:59.188153 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188049 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:59:59.188153 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188102 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:59:59.188341 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188315 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:59:59.188341 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188327 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:59:59.188431 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188344 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:59:59.188431 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188373 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:59:59.188431 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.188379 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kv56z\"" Oct 30 17:59:59.198260 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.198237 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:59:59.199772 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.199467 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:59:59.249909 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.249885 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/956cfd2c-560c-4b7c-8484-1334cf3b10bf-config-out\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250042 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.249916 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250042 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.249945 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956cfd2c-560c-4b7c-8484-1334cf3b10bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250075 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/956cfd2c-560c-4b7c-8484-1334cf3b10bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250102 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250123 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250120 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250224 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250200 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/956cfd2c-560c-4b7c-8484-1334cf3b10bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250224 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250269 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250244 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/956cfd2c-560c-4b7c-8484-1334cf3b10bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250333 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250279 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9st\" (UniqueName: \"kubernetes.io/projected/956cfd2c-560c-4b7c-8484-1334cf3b10bf-kube-api-access-fd9st\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250333 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250315 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-web-config\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250400 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250348 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.250400 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.250371 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.297494 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.297469 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5835e54b-74f0-4ece-a77a-797505233920" path="/var/lib/kubelet/pods/5835e54b-74f0-4ece-a77a-797505233920/volumes" Oct 30 17:59:59.351629 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351606 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/956cfd2c-560c-4b7c-8484-1334cf3b10bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351739 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351664 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351739 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351686 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/956cfd2c-560c-4b7c-8484-1334cf3b10bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351739 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351705 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9st\" (UniqueName: \"kubernetes.io/projected/956cfd2c-560c-4b7c-8484-1334cf3b10bf-kube-api-access-fd9st\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351739 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351730 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-web-config\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351960 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351747 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351960 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351767 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351960 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351798 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/956cfd2c-560c-4b7c-8484-1334cf3b10bf-config-out\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351960 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351843 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351960 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351877 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956cfd2c-560c-4b7c-8484-1334cf3b10bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351960 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351930 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/956cfd2c-560c-4b7c-8484-1334cf3b10bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.351960 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351958 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.352277 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.351980 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.352277 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.352249 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/956cfd2c-560c-4b7c-8484-1334cf3b10bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.352727 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.352480 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/956cfd2c-560c-4b7c-8484-1334cf3b10bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.353118 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.353091 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956cfd2c-560c-4b7c-8484-1334cf3b10bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.354907 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.354850 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/956cfd2c-560c-4b7c-8484-1334cf3b10bf-config-out\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.354907 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.354880 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-web-config\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.355063 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.354938 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.355063 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.354990 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.355063 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.354991 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.355310 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.355292 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.355382 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.355348 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/956cfd2c-560c-4b7c-8484-1334cf3b10bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.355640 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.355618 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.356528 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.356509 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/956cfd2c-560c-4b7c-8484-1334cf3b10bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.368402 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.368383 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9st\" (UniqueName: \"kubernetes.io/projected/956cfd2c-560c-4b7c-8484-1334cf3b10bf-kube-api-access-fd9st\") pod \"alertmanager-main-0\" (UID: \"956cfd2c-560c-4b7c-8484-1334cf3b10bf\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.492651 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.492617 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:59:59.821138 ip-10-0-133-124 kubenswrapper[4582]: I1030 17:59:59.821102 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:59:59.825381 ip-10-0-133-124 kubenswrapper[4582]: W1030 17:59:59.825353 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956cfd2c_560c_4b7c_8484_1334cf3b10bf.slice/crio-d6ef2bc3b6db68457ec567e9b5e3dae45a047cd1a7d93e0d79bcfeafb8f2650e WatchSource:0}: Error finding container d6ef2bc3b6db68457ec567e9b5e3dae45a047cd1a7d93e0d79bcfeafb8f2650e: Status 404 returned error can't find the container with id d6ef2bc3b6db68457ec567e9b5e3dae45a047cd1a7d93e0d79bcfeafb8f2650e Oct 30 18:00:00.807396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:00.807357 4582 generic.go:356] "Generic (PLEG): container finished" podID="956cfd2c-560c-4b7c-8484-1334cf3b10bf" containerID="a174ff5c59a165f54231b92867ed0295b8fdaacdad7c2661c6d15d5e948687f7" exitCode=0 Oct 30 18:00:00.807578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:00.807438 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerDied","Data":"a174ff5c59a165f54231b92867ed0295b8fdaacdad7c2661c6d15d5e948687f7"} Oct 30 18:00:00.807578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:00.807511 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerStarted","Data":"d6ef2bc3b6db68457ec567e9b5e3dae45a047cd1a7d93e0d79bcfeafb8f2650e"} Oct 30 18:00:01.815334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:01.815295 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerStarted","Data":"093c042f0214f5ad6fdff08127dc7d90a2cef821cc65f833c34fcfbe7fea27d8"} Oct 30 18:00:01.815707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:01.815375 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerStarted","Data":"5b807652efa60733f28a5da8ce3a34a0fe7b82d9a561c9f1fcee53fe386ff0b7"} Oct 30 18:00:01.815707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:01.815416 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerStarted","Data":"7a8d9a1c4ab62b06752d2f47925a5dd0daf1d0632f888c314886b098e8caba7f"} Oct 30 18:00:01.815707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:01.815453 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerStarted","Data":"514246acdcaa34c1d92405f9e3e4bd9569a34e0d6ecde44b44a0683c6f60381c"} Oct 30 18:00:01.815707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:01.815488 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerStarted","Data":"383d2b54806aed428a5cd1a4a4276a44f09bfe90434b15630686733b37a5b1da"} Oct 30 18:00:01.815707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:01.815525 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"956cfd2c-560c-4b7c-8484-1334cf3b10bf","Type":"ContainerStarted","Data":"27828ae432afff5283025f9eddd75c7cd8bb40b36c55839283b4c77be19968ac"} Oct 30 18:00:01.886257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:01.886195 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8764294440000002 podStartE2EDuration="2.886177175s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.747731ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:01.885177151 +0000 UTC m=+98.709571468" watchObservedRunningTime="2026-10-30 18:00:01.886177175 +0000 UTC m=+98.710571467" Oct 30 18:00:02.104043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.103944 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:00:02.104324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.104264 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="prometheus" containerID="cri-o://ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f" gracePeriod=600 Oct 30 18:00:02.104324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.104288 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede" gracePeriod=600 Oct 30 18:00:02.104324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.104299 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="config-reloader" containerID="cri-o://595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21" gracePeriod=600 Oct 30 18:00:02.104522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.104363 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy" containerID="cri-o://08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996" gracePeriod=600 Oct 30 18:00:02.104522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.104289 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="thanos-sidecar" containerID="cri-o://0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1" gracePeriod=600 Oct 30 18:00:02.104522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.104339 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-web" containerID="cri-o://d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7" gracePeriod=600 Oct 30 18:00:02.822806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822772 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a10818-9ffe-499d-b728-8b1591912823" containerID="d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede" exitCode=0 Oct 30 18:00:02.822806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822796 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a10818-9ffe-499d-b728-8b1591912823" containerID="08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996" exitCode=0 Oct 30 18:00:02.822806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822802 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a10818-9ffe-499d-b728-8b1591912823" containerID="0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1" exitCode=0 Oct 30 18:00:02.822806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822808 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a10818-9ffe-499d-b728-8b1591912823" containerID="595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21" exitCode=0 Oct 30 18:00:02.822806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822814 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a10818-9ffe-499d-b728-8b1591912823" containerID="ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f" exitCode=0 Oct 30 18:00:02.823311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822842 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede"} Oct 30 18:00:02.823311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822902 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996"} Oct 30 18:00:02.823311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822930 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1"} Oct 30 18:00:02.823311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822953 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21"} Oct 30 18:00:02.823311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:02.822976 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f"} Oct 30 18:00:03.352756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.352732 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.488259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488174 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-kubelet-serving-ca-bundle\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488214 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-1\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488248 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-serving-certs-ca-bundle\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488270 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-metrics-client-ca\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488299 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-config-out\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488329 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-trusted-ca-bundle\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488349 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-kube-api-access-rjkmj\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-kube-api-access-rjkmj\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488371 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-tls-assets\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-tls-assets\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488417 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-metrics-client-certs\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488461 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-thanos-prometheus-http-client-file\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488489 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-grpc-tls\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488522 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488586 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-db\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488620 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-0\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488653 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-kube-rbac-proxy\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488682 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-config\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488712 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-tls\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488706 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-kubelet-serving-ca-bundle" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488756 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-2\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488787 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488787 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-1" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488803 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-trusted-ca-bundle" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488793 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-metrics-client-ca" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:03.488978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.488815 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-web-config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-web-config\") pod \"b5a10818-9ffe-499d-b728-8b1591912823\" (UID: \"b5a10818-9ffe-499d-b728-8b1591912823\") " Oct 30 18:00:03.489537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.489138 4582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-metrics-client-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.489537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.489271 4582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.489537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.489292 4582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.489537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.489306 4582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.489537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.489415 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-serving-certs-ca-bundle" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:03.490194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.489929 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-2" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:03.491553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.491523 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-db" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:03.491660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.491577 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-config-out" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:03.492928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.492898 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-kube-api-access-rjkmj" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "kube-api-access-rjkmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:03.493191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493164 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-thanos-prometheus-http-client-file" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.493279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493242 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-thanos-sidecar-tls" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.493352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493319 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-metrics-client-certs" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.493405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493380 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-tls-assets" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:03.493405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493378 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-0" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:03.493480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493421 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-kube-rbac-proxy-web" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.493878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493851 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-config" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.493985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493894 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-tls" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.493985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.493957 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-kube-rbac-proxy" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.494643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.494624 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-grpc-tls" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.502922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.502900 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-web-config" pod "b5a10818-9ffe-499d-b728-8b1591912823" (UID: "b5a10818-9ffe-499d-b728-8b1591912823"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.590432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590396 4582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590427 4582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590437 4582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-web-config\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590446 4582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590455 4582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-config-out\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590463 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkmj\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-kube-api-access-rjkmj\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590471 4582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5a10818-9ffe-499d-b728-8b1591912823-tls-assets\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590480 4582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-metrics-client-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590489 4582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590498 4582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-grpc-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590506 4582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590514 4582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-db\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590522 4582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5a10818-9ffe-499d-b728-8b1591912823-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590532 4582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-kube-rbac-proxy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590540 4582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-config\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.590652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.590549 4582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5a10818-9ffe-499d-b728-8b1591912823-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.829542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.829448 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5a10818-9ffe-499d-b728-8b1591912823" containerID="d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7" exitCode=0 Oct 30 18:00:03.829542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.829500 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7"} Oct 30 18:00:03.830094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.829551 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.830094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.829600 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5a10818-9ffe-499d-b728-8b1591912823","Type":"ContainerDied","Data":"eced99d37b1420b29a0dfe154eebf8d629b3f53db226cba08cdddde14f9267e9"} Oct 30 18:00:03.830094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.829634 4582 scope.go:122] "RemoveContainer" containerID="d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede" Oct 30 18:00:03.837229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.837082 4582 scope.go:122] "RemoveContainer" containerID="08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996" Oct 30 18:00:03.844689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.844671 4582 scope.go:122] "RemoveContainer" containerID="d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7" Oct 30 18:00:03.851100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.851084 4582 scope.go:122] "RemoveContainer" containerID="0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1" Oct 30 18:00:03.857352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.857331 4582 scope.go:122] "RemoveContainer" containerID="595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21" Oct 30 18:00:03.863848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.863829 4582 scope.go:122] "RemoveContainer" containerID="ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f" Oct 30 18:00:03.870616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.870600 4582 scope.go:122] "RemoveContainer" containerID="d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989" Oct 30 18:00:03.876679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.876657 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:00:03.876738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.876687 4582 scope.go:122] "RemoveContainer" containerID="d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede" Oct 30 18:00:03.876965 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:03.876944 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede\": container with ID starting with d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede not found: ID does not exist" containerID="d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede" Oct 30 18:00:03.877042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.876978 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede"} err="failed to get container status \"d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede\": rpc error: code = NotFound desc = could not find container \"d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede\": container with ID starting with d53361c9e615f79247fa2b8888c52dbf5b249511976ca570f3408c2f10490ede not found: ID does not exist" Oct 30 18:00:03.877042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.877001 4582 scope.go:122] "RemoveContainer" containerID="08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996" Oct 30 18:00:03.877252 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:03.877231 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996\": container with ID starting with 08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996 not found: ID does not exist" containerID="08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996" Oct 30 18:00:03.877289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.877262 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996"} err="failed to get container status \"08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996\": rpc error: code = NotFound desc = could not find container \"08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996\": container with ID starting with 08da568acb0865e2f86855a401a9b5e8a00362b39556ec0284995b7ad90d0996 not found: ID does not exist" Oct 30 18:00:03.877289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.877285 4582 scope.go:122] "RemoveContainer" containerID="d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7" Oct 30 18:00:03.877560 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:03.877540 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7\": container with ID starting with d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7 not found: ID does not exist" containerID="d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7" Oct 30 18:00:03.877599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.877566 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7"} err="failed to get container status \"d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7\": rpc error: code = NotFound desc = could not find container \"d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7\": container with ID starting with d08b67f21cf731c5e6dfcae7571339727f606990581729804e0e91d7feddebf7 not found: ID does not exist" Oct 30 18:00:03.877599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.877581 4582 scope.go:122] "RemoveContainer" containerID="0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1" Oct 30 18:00:03.877814 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:03.877798 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1\": container with ID starting with 0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1 not found: ID does not exist" containerID="0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1" Oct 30 18:00:03.877898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.877866 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1"} err="failed to get container status \"0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1\": rpc error: code = NotFound desc = could not find container \"0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1\": container with ID starting with 0b19a49decf476562d5d76e2d50c499b981b237572485c0049b780556da726d1 not found: ID does not exist" Oct 30 18:00:03.877898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.877882 4582 scope.go:122] "RemoveContainer" containerID="595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21" Oct 30 18:00:03.878118 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:03.878102 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21\": container with ID starting with 595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21 not found: ID does not exist" containerID="595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21" Oct 30 18:00:03.878161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.878122 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21"} err="failed to get container status \"595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21\": rpc error: code = NotFound desc = could not find container \"595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21\": container with ID starting with 595ce633a57346f7a847f4c91802359e430281daafa43fcb40b19117a05c1e21 not found: ID does not exist" Oct 30 18:00:03.878161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.878135 4582 scope.go:122] "RemoveContainer" containerID="ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f" Oct 30 18:00:03.878396 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:03.878364 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f\": container with ID starting with ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f not found: ID does not exist" containerID="ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f" Oct 30 18:00:03.878460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.878405 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f"} err="failed to get container status \"ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f\": rpc error: code = NotFound desc = could not find container \"ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f\": container with ID starting with ace148b4ef13cef01a8119449143f323817abc91a865ef163fe7ee9f12fbac0f not found: ID does not exist" Oct 30 18:00:03.878460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.878426 4582 scope.go:122] "RemoveContainer" containerID="d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989" Oct 30 18:00:03.878657 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:03.878642 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989\": container with ID starting with d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989 not found: ID does not exist" containerID="d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989" Oct 30 18:00:03.878703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.878686 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989"} err="failed to get container status \"d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989\": rpc error: code = NotFound desc = could not find container \"d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989\": container with ID starting with d1a7cbe703f0395592f14284ed2049ad7fe85e8e8aab27afb871a7c5cf1b2989 not found: ID does not exist" Oct 30 18:00:03.886553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.886532 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:00:03.910807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.910784 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:00:03.911156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911139 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="init-config-reloader" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911158 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="init-config-reloader" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911170 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-web" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911177 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-web" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911190 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="thanos-sidecar" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911197 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="thanos-sidecar" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911212 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="config-reloader" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911219 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="config-reloader" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911227 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-thanos" Oct 30 18:00:03.911239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911234 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-thanos" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911251 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911258 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911272 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="prometheus" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911279 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="prometheus" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911356 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-web" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911368 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="thanos-sidecar" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911377 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="config-reloader" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911387 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy-thanos" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911395 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="kube-rbac-proxy" Oct 30 18:00:03.911661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.911404 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a10818-9ffe-499d-b728-8b1591912823" containerName="prometheus" Oct 30 18:00:03.915310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.915294 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.934670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.934648 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:00:03.934755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.934733 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:00:03.934755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.934743 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:00:03.937191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.937176 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:00:03.937767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.937747 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:00:03.947568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947550 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947557 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947577 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947580 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947559 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8go9m9pn67jdm\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947608 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947623 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947634 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x2p2x\"" Oct 30 18:00:03.947668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.947650 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:00:03.957069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.957049 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:00:03.993695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993665 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b967cbbc-55c5-4128-84de-19b842802961-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.993814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993701 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.993814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993720 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.993814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993738 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.993975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993911 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.993975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993938 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.993975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993962 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.993978 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994065 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b967cbbc-55c5-4128-84de-19b842802961-config-out\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994105 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994136 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-web-config\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994160 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994189 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994241 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994269 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994286 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994308 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-config\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994346 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994385 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcjc\" (UniqueName: \"kubernetes.io/projected/b967cbbc-55c5-4128-84de-19b842802961-kube-api-access-kzcjc\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:03.994548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:03.994456 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.031934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.031911 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:00:04.031934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.031923 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:00:04.032813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.032800 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:00:04.095202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095138 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095170 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b967cbbc-55c5-4128-84de-19b842802961-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095194 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095218 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095248 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095283 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095305 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095330 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095346 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095368 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b967cbbc-55c5-4128-84de-19b842802961-config-out\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095607 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.095953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095660 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.095983 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-web-config\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096004 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096023 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096074 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096111 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096154 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096195 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096230 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-config\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096254 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.096339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096297 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcjc\" (UniqueName: \"kubernetes.io/projected/b967cbbc-55c5-4128-84de-19b842802961-kube-api-access-kzcjc\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.097585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.096699 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.097585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.097263 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.097585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.097507 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.098919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.098890 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.099017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.098954 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.099182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.099151 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b967cbbc-55c5-4128-84de-19b842802961-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.099297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.099276 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.099775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.099747 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b967cbbc-55c5-4128-84de-19b842802961-config-out\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.099937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.099917 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.100094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.100072 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.100199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.100178 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.100315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.100291 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.100440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.100424 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.100508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.100435 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-config\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.101204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.101186 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-web-config\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.101262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.101190 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b967cbbc-55c5-4128-84de-19b842802961-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.101454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.101436 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b967cbbc-55c5-4128-84de-19b842802961-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.115144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.115119 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcjc\" (UniqueName: \"kubernetes.io/projected/b967cbbc-55c5-4128-84de-19b842802961-kube-api-access-kzcjc\") pod \"prometheus-k8s-0\" (UID: \"b967cbbc-55c5-4128-84de-19b842802961\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.232450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.232409 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:04.562301 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:00:04.562269 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb967cbbc_55c5_4128_84de_19b842802961.slice/crio-4ddf7980b1aace4aa72335a1709ea92bf3bf0f8053b2d3be60a8fe638c3a011d WatchSource:0}: Error finding container 4ddf7980b1aace4aa72335a1709ea92bf3bf0f8053b2d3be60a8fe638c3a011d: Status 404 returned error can't find the container with id 4ddf7980b1aace4aa72335a1709ea92bf3bf0f8053b2d3be60a8fe638c3a011d Oct 30 18:00:04.569577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.569558 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:00:04.837001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.836921 4582 generic.go:356] "Generic (PLEG): container finished" podID="b967cbbc-55c5-4128-84de-19b842802961" containerID="9d09e9e08f8d8e1c8dcd70d66a50d98b5cb8f7d292308c07dd155f6cba3f4ade" exitCode=0 Oct 30 18:00:04.837388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.837007 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerDied","Data":"9d09e9e08f8d8e1c8dcd70d66a50d98b5cb8f7d292308c07dd155f6cba3f4ade"} Oct 30 18:00:04.837388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.837087 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerStarted","Data":"4ddf7980b1aace4aa72335a1709ea92bf3bf0f8053b2d3be60a8fe638c3a011d"} Oct 30 18:00:04.973540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:04.973508 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:05.077204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.077171 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:05.300885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.300854 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a10818-9ffe-499d-b728-8b1591912823" path="/var/lib/kubelet/pods/b5a10818-9ffe-499d-b728-8b1591912823/volumes" Oct 30 18:00:05.844436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.844403 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerStarted","Data":"92e081f724a9a56986601db7ef6838027b4e4ca41add52517d4b01a912c648f2"} Oct 30 18:00:05.844910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.844462 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerStarted","Data":"0d66e608601df52567983aa8e9686578061eff1c32938397831d0da09bad022f"} Oct 30 18:00:05.844910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.844488 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerStarted","Data":"d580a4f8da06b417b748942152b532d35302b84b8da030b8d75ced3951a64563"} Oct 30 18:00:05.844910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.844515 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerStarted","Data":"49c1034a8105905c77c30e48c44b8fa0745237b74a435ba8987b862e76bac0d0"} Oct 30 18:00:05.844910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.844539 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerStarted","Data":"a5e2dcda1e4636dff4229d7bb1678aecb231e19c96c649ebc20383f8953de3e3"} Oct 30 18:00:05.844910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.844566 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b967cbbc-55c5-4128-84de-19b842802961","Type":"ContainerStarted","Data":"3c9f097678e16cf194492dde7fde4ab31ad5b1e76c03cf685e0e8a0461af310d"} Oct 30 18:00:05.899575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:05.899522 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.878378031 podStartE2EDuration="2.899507931s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.1299ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:05.897546314 +0000 UTC m=+102.721940605" watchObservedRunningTime="2026-10-30 18:00:05.899507931 +0000 UTC m=+102.723902223" Oct 30 18:00:06.223452 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:06.223420 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:00:09.233290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:09.233251 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:00:11.531957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:11.531909 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:18.799964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:18.799797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:18.901933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:18.901896 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:19.002002 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:19.001970 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:35.954888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:35.954846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:36.175346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:36.175310 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:36.226286 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:00:36.223957 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:00:37.595309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:37.595271 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:56.226656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:56.226624 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:56.321914 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:00:56.321881 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:04.233417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:01:04.233380 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:04.248785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:01:04.248761 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:05.073235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:01:05.073207 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:06.223031 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:01:06.222999 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:01:11.645066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:01:11.645032 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:28.194902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:01:28.194808 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:36.222984 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:01:36.222946 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:02:06.225108 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:02:06.225030 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:02:11.769024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:11.768958 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:28.513504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:28.513471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:28.635501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:28.635453 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:28.746294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:28.746256 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:29.956568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:29.956531 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:30.052526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:30.052476 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:31.051802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:31.051763 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:33.824446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:33.824410 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:33.924926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:33.924892 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:34.151797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:34.151760 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:34.322396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:34.322357 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:34.601291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:34.601220 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:36.223667 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:02:36.223636 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:02:38.061739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:02:38.061707 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:06.223088 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:03:06.223049 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:03:11.869176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:11.869141 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:12.733189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:12.733143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:16.334648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:16.334614 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:16.751988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:16.751952 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:16.853942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:16.853908 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:19.527010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.526926 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg"] Oct 30 18:03:19.530370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.530348 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:19.555457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.555439 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 18:03:19.555589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.555476 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:03:19.555589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.555545 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-w44qq\"" Oct 30 18:03:19.556742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.556724 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg"] Oct 30 18:03:19.669563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.669526 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskdq\" (UniqueName: \"kubernetes.io/projected/ba2a2861-1aab-4a42-907b-ebc0e099e8c4-kube-api-access-kskdq\") pod \"jobset-operator-674cd59dc5-9hlhg\" (UID: \"ba2a2861-1aab-4a42-907b-ebc0e099e8c4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:19.669765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.669572 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba2a2861-1aab-4a42-907b-ebc0e099e8c4-tmp\") pod \"jobset-operator-674cd59dc5-9hlhg\" (UID: \"ba2a2861-1aab-4a42-907b-ebc0e099e8c4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:19.727759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.727712 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:19.771105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.771073 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskdq\" (UniqueName: \"kubernetes.io/projected/ba2a2861-1aab-4a42-907b-ebc0e099e8c4-kube-api-access-kskdq\") pod \"jobset-operator-674cd59dc5-9hlhg\" (UID: \"ba2a2861-1aab-4a42-907b-ebc0e099e8c4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:19.771182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.771110 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba2a2861-1aab-4a42-907b-ebc0e099e8c4-tmp\") pod \"jobset-operator-674cd59dc5-9hlhg\" (UID: \"ba2a2861-1aab-4a42-907b-ebc0e099e8c4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:19.771542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.771524 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba2a2861-1aab-4a42-907b-ebc0e099e8c4-tmp\") pod \"jobset-operator-674cd59dc5-9hlhg\" (UID: \"ba2a2861-1aab-4a42-907b-ebc0e099e8c4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:19.787930 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.787869 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskdq\" (UniqueName: \"kubernetes.io/projected/ba2a2861-1aab-4a42-907b-ebc0e099e8c4-kube-api-access-kskdq\") pod \"jobset-operator-674cd59dc5-9hlhg\" (UID: \"ba2a2861-1aab-4a42-907b-ebc0e099e8c4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:19.861391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:19.861356 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" Oct 30 18:03:20.152035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:20.152005 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg"] Oct 30 18:03:20.157011 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:03:20.156982 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba2a2861_1aab_4a42_907b_ebc0e099e8c4.slice/crio-928926b5c9f7f6850232f328d9fe7eee2c3d9ffbd9d53d3834c73977bc99e0e5 WatchSource:0}: Error finding container 928926b5c9f7f6850232f328d9fe7eee2c3d9ffbd9d53d3834c73977bc99e0e5: Status 404 returned error can't find the container with id 928926b5c9f7f6850232f328d9fe7eee2c3d9ffbd9d53d3834c73977bc99e0e5 Oct 30 18:03:20.517842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:20.517753 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:20.566012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:20.565974 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" event={"ID":"ba2a2861-1aab-4a42-907b-ebc0e099e8c4","Type":"ContainerStarted","Data":"928926b5c9f7f6850232f328d9fe7eee2c3d9ffbd9d53d3834c73977bc99e0e5"} Oct 30 18:03:23.579030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:23.578993 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" event={"ID":"ba2a2861-1aab-4a42-907b-ebc0e099e8c4","Type":"ContainerStarted","Data":"4b2217ee500e9316550fe4ffb677e2d197f84d5298967d1f0b557ba986061a57"} Oct 30 18:03:23.632100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:23.632045 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9hlhg" podStartSLOduration=1.988972593 podStartE2EDuration="4.632030887s" totalImagesPullingTime="2.643058294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:23.63107649 +0000 UTC m=+300.455470781" watchObservedRunningTime="2026-10-30 18:03:23.632030887 +0000 UTC m=+300.456425178" Oct 30 18:03:24.724881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:24.724848 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:25.353423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:25.353381 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:25.477696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:25.477658 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:25.577803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:25.577771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:25.708866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:25.708816 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:26.329682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.329648 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:26.630354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.630322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:26.779627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.779595 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt"] Oct 30 18:03:26.783186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.783168 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.815044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.815011 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt"] Oct 30 18:03:26.827233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.827208 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.827346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.827240 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.827346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.827269 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.827417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.827376 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.827453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.827414 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjlq\" (UniqueName: \"kubernetes.io/projected/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-kube-api-access-dqjlq\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.827509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.827493 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.879410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.879358 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:03:26.879745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.879417 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:03:26.879745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.879702 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:03:26.879860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.879753 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:03:26.879860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.879765 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7dbhx\"" Oct 30 18:03:26.879938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.879904 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:03:26.904690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.904625 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:26.928539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.928507 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjlq\" (UniqueName: \"kubernetes.io/projected/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-kube-api-access-dqjlq\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.928666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.928555 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.928666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.928583 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.928666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.928608 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.928666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.928634 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.928666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.928659 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.929073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.929053 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.931117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.931091 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.931207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.931153 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.931323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.931303 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.931360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.931350 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:26.945708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:26.945680 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjlq\" (UniqueName: \"kubernetes.io/projected/bdc8ff88-1234-4af3-81c5-9dbeaa05df9d-kube-api-access-dqjlq\") pod \"opendatahub-operator-controller-manager-58c455bdd6-cd8zt\" (UID: \"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:27.001052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.001025 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:27.097881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.097859 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:27.140706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.140679 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:27.252414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.252339 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:27.362038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.362006 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:27.441320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.441286 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt"] Oct 30 18:03:27.445704 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:03:27.445679 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc8ff88_1234_4af3_81c5_9dbeaa05df9d.slice/crio-f7c9e74db9b94e083a6dbf4d4cc8d5034be6ea90909a84dd1b91a973d2dabc78 WatchSource:0}: Error finding container f7c9e74db9b94e083a6dbf4d4cc8d5034be6ea90909a84dd1b91a973d2dabc78: Status 404 returned error can't find the container with id f7c9e74db9b94e083a6dbf4d4cc8d5034be6ea90909a84dd1b91a973d2dabc78 Oct 30 18:03:27.447466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.447450 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:03:27.471388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.471365 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:27.596339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:27.596266 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" event={"ID":"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d","Type":"ContainerStarted","Data":"f7c9e74db9b94e083a6dbf4d4cc8d5034be6ea90909a84dd1b91a973d2dabc78"} Oct 30 18:03:28.610773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:28.610737 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:28.710882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:28.710846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:28.928701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:28.928661 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:29.560038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:29.559997 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:30.611334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:30.611241 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" event={"ID":"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d","Type":"ContainerStarted","Data":"25bfed72c214463a945c7c11aa0337b81e98a9e7f02eb375883b14fb81edb53f"} Oct 30 18:03:31.229663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:31.229629 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:31.617028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:31.616942 4582 generic.go:356] "Generic (PLEG): container finished" podID="bdc8ff88-1234-4af3-81c5-9dbeaa05df9d" containerID="25bfed72c214463a945c7c11aa0337b81e98a9e7f02eb375883b14fb81edb53f" exitCode=0 Oct 30 18:03:31.617392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:31.617033 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" event={"ID":"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d","Type":"ContainerDied","Data":"25bfed72c214463a945c7c11aa0337b81e98a9e7f02eb375883b14fb81edb53f"} Oct 30 18:03:32.338454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:32.338382 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:32.623301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:32.623266 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" event={"ID":"bdc8ff88-1234-4af3-81c5-9dbeaa05df9d","Type":"ContainerStarted","Data":"25b12c00fbde4d84f3a98046290d047a7ff9ea833736663f561fb541ae0dccd5"} Oct 30 18:03:32.623684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:32.623355 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:32.699877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:32.699797 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" podStartSLOduration=3.0001532 podStartE2EDuration="6.699783792s" totalImagesPullingTime="3.229103897s" totalInitContainerRuntime="470.526695ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:32.689631365 +0000 UTC m=+309.514025657" watchObservedRunningTime="2026-10-30 18:03:32.699783792 +0000 UTC m=+309.524178084" Oct 30 18:03:33.910720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:33.910685 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:34.083566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:34.083529 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:34.183916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:34.183807 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:36.131395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:36.131357 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:36.223360 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:03:36.223326 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:03:36.231980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:36.231952 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:38.445478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.445442 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:38.543702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.543669 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:38.702782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.702702 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27"] Oct 30 18:03:38.709192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.709173 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:38.724903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.724880 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27"] Oct 30 18:03:38.733295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.733272 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:03:38.733421 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.733371 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:03:38.733421 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.733396 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-h8bg9\"" Oct 30 18:03:38.833704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.833674 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13292340-3a3f-4a46-90ae-cd5a5670350e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ghw27\" (UID: \"13292340-3a3f-4a46-90ae-cd5a5670350e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:38.833849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.833706 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jbs\" (UniqueName: \"kubernetes.io/projected/13292340-3a3f-4a46-90ae-cd5a5670350e-kube-api-access-b7jbs\") pod \"cert-manager-webhook-5dbbbf75cc-ghw27\" (UID: \"13292340-3a3f-4a46-90ae-cd5a5670350e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:38.934422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.934389 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jbs\" (UniqueName: \"kubernetes.io/projected/13292340-3a3f-4a46-90ae-cd5a5670350e-kube-api-access-b7jbs\") pod \"cert-manager-webhook-5dbbbf75cc-ghw27\" (UID: \"13292340-3a3f-4a46-90ae-cd5a5670350e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:38.934554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.934479 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13292340-3a3f-4a46-90ae-cd5a5670350e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ghw27\" (UID: \"13292340-3a3f-4a46-90ae-cd5a5670350e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:38.951617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.951587 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13292340-3a3f-4a46-90ae-cd5a5670350e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ghw27\" (UID: \"13292340-3a3f-4a46-90ae-cd5a5670350e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:38.951725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:38.951623 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jbs\" (UniqueName: \"kubernetes.io/projected/13292340-3a3f-4a46-90ae-cd5a5670350e-kube-api-access-b7jbs\") pod \"cert-manager-webhook-5dbbbf75cc-ghw27\" (UID: \"13292340-3a3f-4a46-90ae-cd5a5670350e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:39.020636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:39.020587 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:39.313515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:39.313435 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27"] Oct 30 18:03:39.318207 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:03:39.318182 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13292340_3a3f_4a46_90ae_cd5a5670350e.slice/crio-e57aa666b45deebc164d73da94b2f8f5d0b0aa1100032ff395950b99a5c07376 WatchSource:0}: Error finding container e57aa666b45deebc164d73da94b2f8f5d0b0aa1100032ff395950b99a5c07376: Status 404 returned error can't find the container with id e57aa666b45deebc164d73da94b2f8f5d0b0aa1100032ff395950b99a5c07376 Oct 30 18:03:39.529519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:39.529491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:39.629876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:39.629848 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:39.652326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:39.652297 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" event={"ID":"13292340-3a3f-4a46-90ae-cd5a5670350e","Type":"ContainerStarted","Data":"e57aa666b45deebc164d73da94b2f8f5d0b0aa1100032ff395950b99a5c07376"} Oct 30 18:03:40.548261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.546649 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf"] Oct 30 18:03:40.550307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.550273 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:40.554741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.553976 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.565221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.565176 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf"] Oct 30 18:03:40.573409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.573337 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 18:03:40.573409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.573340 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:03:40.573409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.573381 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 18:03:40.573618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.573508 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ks7zb\"" Oct 30 18:03:40.575973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.575938 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:03:40.651088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.651049 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e0ef342-18fe-4c80-9911-00af0f03669b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.651256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.651097 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e0ef342-18fe-4c80-9911-00af0f03669b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.651256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.651171 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggdz\" (UniqueName: \"kubernetes.io/projected/3e0ef342-18fe-4c80-9911-00af0f03669b-kube-api-access-mggdz\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.686076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.686040 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:40.752349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.752312 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e0ef342-18fe-4c80-9911-00af0f03669b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.752508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.752355 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e0ef342-18fe-4c80-9911-00af0f03669b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.752508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.752406 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggdz\" (UniqueName: \"kubernetes.io/projected/3e0ef342-18fe-4c80-9911-00af0f03669b-kube-api-access-mggdz\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.752807 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:03:40.752788 4582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Oct 30 18:03:40.752893 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:03:40.752882 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0ef342-18fe-4c80-9911-00af0f03669b-plugin-serving-cert podName:3e0ef342-18fe-4c80-9911-00af0f03669b nodeName:}" failed. No retries permitted until 2026-10-30 18:03:41.252862025 +0000 UTC m=+318.077256301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3e0ef342-18fe-4c80-9911-00af0f03669b-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-dhkgf" (UID: "3e0ef342-18fe-4c80-9911-00af0f03669b") : secret "plugin-serving-cert" not found Oct 30 18:03:40.753682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.753658 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e0ef342-18fe-4c80-9911-00af0f03669b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.780397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.780366 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggdz\" (UniqueName: \"kubernetes.io/projected/3e0ef342-18fe-4c80-9911-00af0f03669b-kube-api-access-mggdz\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:40.855353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.855258 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:40.955438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:40.955408 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:41.255694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:41.255652 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e0ef342-18fe-4c80-9911-00af0f03669b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:41.257996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:41.257968 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e0ef342-18fe-4c80-9911-00af0f03669b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dhkgf\" (UID: \"3e0ef342-18fe-4c80-9911-00af0f03669b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:41.472879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:41.472846 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" Oct 30 18:03:41.772124 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:41.772089 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf"] Oct 30 18:03:41.777080 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:03:41.777057 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e0ef342_18fe_4c80_9911_00af0f03669b.slice/crio-8546e7524f8ec782420265fa03a5a352442fa249a846bdf7e037299a52052257 WatchSource:0}: Error finding container 8546e7524f8ec782420265fa03a5a352442fa249a846bdf7e037299a52052257: Status 404 returned error can't find the container with id 8546e7524f8ec782420265fa03a5a352442fa249a846bdf7e037299a52052257 Oct 30 18:03:42.677629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:42.677590 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" event={"ID":"13292340-3a3f-4a46-90ae-cd5a5670350e","Type":"ContainerStarted","Data":"2061ac2a3d7a804eb437bd196bf9b47d40b6052792f7323e5e0b5dc290fde595"} Oct 30 18:03:42.677845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:42.677680 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:42.679356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:42.679329 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" event={"ID":"3e0ef342-18fe-4c80-9911-00af0f03669b","Type":"ContainerStarted","Data":"8546e7524f8ec782420265fa03a5a352442fa249a846bdf7e037299a52052257"} Oct 30 18:03:42.713267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:42.713213 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" podStartSLOduration=1.824800134 podStartE2EDuration="4.713195619s" totalImagesPullingTime="2.888395485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:42.711531407 +0000 UTC m=+319.535925730" watchObservedRunningTime="2026-10-30 18:03:42.713195619 +0000 UTC m=+319.537589917" Oct 30 18:03:43.162004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:43.161959 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:43.262672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:43.262636 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:43.363262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:43.363230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:43.630229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:43.630200 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-cd8zt" Oct 30 18:03:44.951152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:44.951111 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k"] Oct 30 18:03:44.978451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:44.978425 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k"] Oct 30 18:03:44.978613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:44.978519 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.000569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.000540 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xbkzg\"" Oct 30 18:03:45.092898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.092866 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113aefb7-b0bd-4389-8e73-24e3f99be7ac-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6mt6k\" (UID: \"113aefb7-b0bd-4389-8e73-24e3f99be7ac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.093074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.092915 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwzm\" (UniqueName: \"kubernetes.io/projected/113aefb7-b0bd-4389-8e73-24e3f99be7ac-kube-api-access-zkwzm\") pod \"cert-manager-cainjector-5c57bbd4fc-6mt6k\" (UID: \"113aefb7-b0bd-4389-8e73-24e3f99be7ac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.155621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.155583 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:45.193647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.193616 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113aefb7-b0bd-4389-8e73-24e3f99be7ac-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6mt6k\" (UID: \"113aefb7-b0bd-4389-8e73-24e3f99be7ac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.193835 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.193664 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwzm\" (UniqueName: \"kubernetes.io/projected/113aefb7-b0bd-4389-8e73-24e3f99be7ac-kube-api-access-zkwzm\") pod \"cert-manager-cainjector-5c57bbd4fc-6mt6k\" (UID: \"113aefb7-b0bd-4389-8e73-24e3f99be7ac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.212444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.212377 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113aefb7-b0bd-4389-8e73-24e3f99be7ac-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6mt6k\" (UID: \"113aefb7-b0bd-4389-8e73-24e3f99be7ac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.213780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.213754 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwzm\" (UniqueName: \"kubernetes.io/projected/113aefb7-b0bd-4389-8e73-24e3f99be7ac-kube-api-access-zkwzm\") pod \"cert-manager-cainjector-5c57bbd4fc-6mt6k\" (UID: \"113aefb7-b0bd-4389-8e73-24e3f99be7ac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.272562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.272524 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:45.294634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.294608 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" Oct 30 18:03:45.606734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.606647 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k"] Oct 30 18:03:45.613347 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:03:45.613315 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod113aefb7_b0bd_4389_8e73_24e3f99be7ac.slice/crio-2bb82e760588a0d59c2e5c47d2e15eb97c27a77490c6e170a386fdbb510739bc WatchSource:0}: Error finding container 2bb82e760588a0d59c2e5c47d2e15eb97c27a77490c6e170a386fdbb510739bc: Status 404 returned error can't find the container with id 2bb82e760588a0d59c2e5c47d2e15eb97c27a77490c6e170a386fdbb510739bc Oct 30 18:03:45.695764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.695727 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" event={"ID":"113aefb7-b0bd-4389-8e73-24e3f99be7ac","Type":"ContainerStarted","Data":"52393a4798a741f21cb32375572fcd203bd47f06d3004ee02beab4a51ee9d248"} Oct 30 18:03:45.695910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.695778 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" event={"ID":"113aefb7-b0bd-4389-8e73-24e3f99be7ac","Type":"ContainerStarted","Data":"2bb82e760588a0d59c2e5c47d2e15eb97c27a77490c6e170a386fdbb510739bc"} Oct 30 18:03:45.799729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.799687 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:45.902317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:45.902283 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:46.643710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:46.643680 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:46.701377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:46.701338 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" event={"ID":"3e0ef342-18fe-4c80-9911-00af0f03669b","Type":"ContainerStarted","Data":"3ffef5668b1492147093040f0e1b3dac6e72cae5d96ed20bc6637e60f0ea08a1"} Oct 30 18:03:46.734754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:46.734707 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dhkgf" podStartSLOduration=2.353014514 podStartE2EDuration="6.734693945s" totalImagesPullingTime="4.381679431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:46.734618387 +0000 UTC m=+323.559012690" watchObservedRunningTime="2026-10-30 18:03:46.734693945 +0000 UTC m=+323.559088236" Oct 30 18:03:46.735541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:46.735506 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6mt6k" podStartSLOduration=2.735498963 podStartE2EDuration="2.735498963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:45.740711349 +0000 UTC m=+322.565105670" watchObservedRunningTime="2026-10-30 18:03:46.735498963 +0000 UTC m=+323.559893259" Oct 30 18:03:48.295720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.295693 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:48.686779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.686749 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" Oct 30 18:03:48.847120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.847078 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-t2654"] Oct 30 18:03:48.852367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.852347 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:48.867675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.867650 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-t2654"] Oct 30 18:03:48.869744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.869725 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9q2q6\"" Oct 30 18:03:48.927633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.927608 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1820228-9536-45be-b7da-b4673c188120-bound-sa-token\") pod \"cert-manager-597b95cb84-t2654\" (UID: \"f1820228-9536-45be-b7da-b4673c188120\") " pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:48.927879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:48.927654 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4dbv\" (UniqueName: \"kubernetes.io/projected/f1820228-9536-45be-b7da-b4673c188120-kube-api-access-t4dbv\") pod \"cert-manager-597b95cb84-t2654\" (UID: \"f1820228-9536-45be-b7da-b4673c188120\") " pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:49.029134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.029064 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1820228-9536-45be-b7da-b4673c188120-bound-sa-token\") pod \"cert-manager-597b95cb84-t2654\" (UID: \"f1820228-9536-45be-b7da-b4673c188120\") " pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:49.029134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.029108 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4dbv\" (UniqueName: \"kubernetes.io/projected/f1820228-9536-45be-b7da-b4673c188120-kube-api-access-t4dbv\") pod \"cert-manager-597b95cb84-t2654\" (UID: \"f1820228-9536-45be-b7da-b4673c188120\") " pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:49.046077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.046046 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4dbv\" (UniqueName: \"kubernetes.io/projected/f1820228-9536-45be-b7da-b4673c188120-kube-api-access-t4dbv\") pod \"cert-manager-597b95cb84-t2654\" (UID: \"f1820228-9536-45be-b7da-b4673c188120\") " pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:49.046342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.046321 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1820228-9536-45be-b7da-b4673c188120-bound-sa-token\") pod \"cert-manager-597b95cb84-t2654\" (UID: \"f1820228-9536-45be-b7da-b4673c188120\") " pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:49.163023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.162981 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-t2654" Oct 30 18:03:49.451428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.451390 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-t2654"] Oct 30 18:03:49.456340 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:03:49.456315 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1820228_9536_45be_b7da_b4673c188120.slice/crio-be146896965b3d60f5cff1d447b7b68d8acc49d6b44c4fdfb2955c1dff11d36c WatchSource:0}: Error finding container be146896965b3d60f5cff1d447b7b68d8acc49d6b44c4fdfb2955c1dff11d36c: Status 404 returned error can't find the container with id be146896965b3d60f5cff1d447b7b68d8acc49d6b44c4fdfb2955c1dff11d36c Oct 30 18:03:49.721684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.721605 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-t2654" event={"ID":"f1820228-9536-45be-b7da-b4673c188120","Type":"ContainerStarted","Data":"ed5869fb0835b887f4842d865d3113203e6e17453abd89ffec1db55e1f27cd5e"} Oct 30 18:03:49.721684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.721644 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-t2654" event={"ID":"f1820228-9536-45be-b7da-b4673c188120","Type":"ContainerStarted","Data":"be146896965b3d60f5cff1d447b7b68d8acc49d6b44c4fdfb2955c1dff11d36c"} Oct 30 18:03:49.756745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:49.756688 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-t2654" podStartSLOduration=1.756675116 podStartE2EDuration="1.756675116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:49.75571989 +0000 UTC m=+326.580114184" watchObservedRunningTime="2026-10-30 18:03:49.756675116 +0000 UTC m=+326.581069408" Oct 30 18:03:50.549979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:50.549945 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:50.662997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:50.662967 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:50.765038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:50.765009 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:50.930582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:50.930551 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:51.051231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:51.051200 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:57.148148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:57.148108 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:57.249023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:03:57.248986 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:00.152966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:00.152938 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:00.261634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:00.261603 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:01.312287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:01.312254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:01.576709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:01.576622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:01.810850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:01.810805 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:02.102908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.102862 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:02.151154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.151126 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6"] Oct 30 18:04:02.154033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.154005 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.170626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.170605 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:04:02.170739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.170666 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:04:02.187077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.187050 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6"] Oct 30 18:04:02.245344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.245312 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6299316-1bf3-4b66-99fa-55fcd37b64f2-tls-certs\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.245511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.245355 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6299316-1bf3-4b66-99fa-55fcd37b64f2-tmp\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.245511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.245382 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7sld\" (UniqueName: \"kubernetes.io/projected/d6299316-1bf3-4b66-99fa-55fcd37b64f2-kube-api-access-d7sld\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.258834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.258798 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:04:02.258956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.258836 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:04:02.259006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.258957 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2c4ww\"" Oct 30 18:04:02.301364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.301328 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:02.346519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.346490 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7sld\" (UniqueName: \"kubernetes.io/projected/d6299316-1bf3-4b66-99fa-55fcd37b64f2-kube-api-access-d7sld\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.346943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.346606 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6299316-1bf3-4b66-99fa-55fcd37b64f2-tls-certs\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.346943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.346672 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6299316-1bf3-4b66-99fa-55fcd37b64f2-tmp\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.349147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.349123 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6299316-1bf3-4b66-99fa-55fcd37b64f2-tmp\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.349561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.349538 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6299316-1bf3-4b66-99fa-55fcd37b64f2-tls-certs\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.372152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.372122 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7sld\" (UniqueName: \"kubernetes.io/projected/d6299316-1bf3-4b66-99fa-55fcd37b64f2-kube-api-access-d7sld\") pod \"kube-auth-proxy-7ffc6d98d7-pkgv6\" (UID: \"d6299316-1bf3-4b66-99fa-55fcd37b64f2\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.465903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.465872 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" Oct 30 18:04:02.500736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.500709 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:02.685672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.685596 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:02.755451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.755419 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6"] Oct 30 18:04:02.760103 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:04:02.760079 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6299316_1bf3_4b66_99fa_55fcd37b64f2.slice/crio-b88008e9b601f8c5db53dbcad8b8b723d6a79aef8226ea85c5211885e5fa0072 WatchSource:0}: Error finding container b88008e9b601f8c5db53dbcad8b8b723d6a79aef8226ea85c5211885e5fa0072: Status 404 returned error can't find the container with id b88008e9b601f8c5db53dbcad8b8b723d6a79aef8226ea85c5211885e5fa0072 Oct 30 18:04:02.774303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.774276 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" event={"ID":"d6299316-1bf3-4b66-99fa-55fcd37b64f2","Type":"ContainerStarted","Data":"b88008e9b601f8c5db53dbcad8b8b723d6a79aef8226ea85c5211885e5fa0072"} Oct 30 18:04:02.964870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:02.964787 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:03.073344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:03.073309 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:03.749482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:03.749438 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:04.524611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:04.524577 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:05.149437 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:05.149405 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:05.252734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:05.252649 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:05.351127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:05.351097 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:05.436878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:05.436847 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:05.790599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:05.790565 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" event={"ID":"d6299316-1bf3-4b66-99fa-55fcd37b64f2","Type":"ContainerStarted","Data":"673314ff3dc38d8fae5497d35b851a150bb4d6594986382111c61499e6a84141"} Oct 30 18:04:05.825744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:05.825697 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-pkgv6" podStartSLOduration=1.606148444 podStartE2EDuration="3.825684124s" totalImagesPullingTime="2.21953568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:05.824027858 +0000 UTC m=+342.648422149" watchObservedRunningTime="2026-10-30 18:04:05.825684124 +0000 UTC m=+342.650078415" Oct 30 18:04:06.224183 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:04:06.224147 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:04:10.304212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:10.304180 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:10.403837 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:10.403799 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:11.973356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:11.973322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:14.154931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.154901 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:14.256321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.256289 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:14.803784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.803753 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p"] Oct 30 18:04:14.874855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.874806 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p"] Oct 30 18:04:14.875012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.874920 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:14.890151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.890129 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 18:04:14.900034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.900016 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 18:04:14.900114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.900061 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 18:04:14.900152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.900116 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4mwn6\"" Oct 30 18:04:14.958711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.958681 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afaf6c79-4f4d-4b09-9357-2fba668547a7-cert\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:14.958906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.958718 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfxr\" (UniqueName: \"kubernetes.io/projected/afaf6c79-4f4d-4b09-9357-2fba668547a7-kube-api-access-tbfxr\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:14.958906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.958756 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afaf6c79-4f4d-4b09-9357-2fba668547a7-metrics-certs\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:14.958906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:14.958810 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afaf6c79-4f4d-4b09-9357-2fba668547a7-manager-config\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.059484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.059411 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afaf6c79-4f4d-4b09-9357-2fba668547a7-metrics-certs\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.059484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.059440 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afaf6c79-4f4d-4b09-9357-2fba668547a7-manager-config\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.059484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.059486 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afaf6c79-4f4d-4b09-9357-2fba668547a7-cert\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.059762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.059503 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfxr\" (UniqueName: \"kubernetes.io/projected/afaf6c79-4f4d-4b09-9357-2fba668547a7-kube-api-access-tbfxr\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.060252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.060214 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afaf6c79-4f4d-4b09-9357-2fba668547a7-manager-config\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.062132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.062067 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afaf6c79-4f4d-4b09-9357-2fba668547a7-metrics-certs\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.062132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.062125 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afaf6c79-4f4d-4b09-9357-2fba668547a7-cert\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.079912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.079884 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfxr\" (UniqueName: \"kubernetes.io/projected/afaf6c79-4f4d-4b09-9357-2fba668547a7-kube-api-access-tbfxr\") pod \"jobset-controller-manager-84f5bbc5c6-6rr5p\" (UID: \"afaf6c79-4f4d-4b09-9357-2fba668547a7\") " pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.198209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.198177 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:15.511091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.511053 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p"] Oct 30 18:04:15.515884 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:04:15.515844 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafaf6c79_4f4d_4b09_9357_2fba668547a7.slice/crio-61c7c67c8aff4144193aebb4240b0249fe8076a6fafdb309c5278fad3fe502ae WatchSource:0}: Error finding container 61c7c67c8aff4144193aebb4240b0249fe8076a6fafdb309c5278fad3fe502ae: Status 404 returned error can't find the container with id 61c7c67c8aff4144193aebb4240b0249fe8076a6fafdb309c5278fad3fe502ae Oct 30 18:04:15.835056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:15.834981 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" event={"ID":"afaf6c79-4f4d-4b09-9357-2fba668547a7","Type":"ContainerStarted","Data":"61c7c67c8aff4144193aebb4240b0249fe8076a6fafdb309c5278fad3fe502ae"} Oct 30 18:04:18.852505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:18.852425 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" event={"ID":"afaf6c79-4f4d-4b09-9357-2fba668547a7","Type":"ContainerStarted","Data":"1893755aa039daafda7073858117716cfcf03a8eccb44c63c9e3718546ab030f"} Oct 30 18:04:18.852867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:18.852564 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:18.893634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:18.893588 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" podStartSLOduration=1.967860418 podStartE2EDuration="4.893575413s" totalImagesPullingTime="2.925714995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:18.89271583 +0000 UTC m=+355.717110121" watchObservedRunningTime="2026-10-30 18:04:18.893575413 +0000 UTC m=+355.717969705" Oct 30 18:04:21.558302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:21.558266 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:21.676693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:21.676662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:21.778651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:21.778621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:22.496324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:22.496291 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:22.595974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:22.595943 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:29.862814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:29.862780 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-84f5bbc5c6-6rr5p" Oct 30 18:04:36.222810 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:04:36.222777 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:04:39.543158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:39.543124 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:39.640446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:39.640415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:40.286908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:04:40.286878 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:00.621451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:00.621370 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:00.836450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:00.836415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:01.031083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:01.031002 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:01.173664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:01.173628 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:02.279326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.279298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:02.429262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.429230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:02.568854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.568764 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:02.608375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.608342 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sbh4k"] Oct 30 18:05:02.612373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.612353 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:02.633801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.633763 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:05:02.643210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.643189 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:05:02.643352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.643223 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 18:05:02.643424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.643224 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wdq9c\"" Oct 30 18:05:02.644803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.644784 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sbh4k"] Oct 30 18:05:02.677861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.677816 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjtd\" (UniqueName: \"kubernetes.io/projected/e141d889-3d52-4eb2-8eb5-6214d5078068-kube-api-access-5tjtd\") pod \"kserve-controller-manager-776cc66d47-sbh4k\" (UID: \"e141d889-3d52-4eb2-8eb5-6214d5078068\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:02.677978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.677871 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e141d889-3d52-4eb2-8eb5-6214d5078068-cert\") pod \"kserve-controller-manager-776cc66d47-sbh4k\" (UID: \"e141d889-3d52-4eb2-8eb5-6214d5078068\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:02.708197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.708163 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-59st9"] Oct 30 18:05:02.711671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.711652 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:02.728140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.728114 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:02.733985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.733971 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 18:05:02.734082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.734024 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4946q\"" Oct 30 18:05:02.741338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.741313 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-59st9"] Oct 30 18:05:02.778371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.778346 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjtd\" (UniqueName: \"kubernetes.io/projected/e141d889-3d52-4eb2-8eb5-6214d5078068-kube-api-access-5tjtd\") pod \"kserve-controller-manager-776cc66d47-sbh4k\" (UID: \"e141d889-3d52-4eb2-8eb5-6214d5078068\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:02.778473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.778376 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e141d889-3d52-4eb2-8eb5-6214d5078068-cert\") pod \"kserve-controller-manager-776cc66d47-sbh4k\" (UID: \"e141d889-3d52-4eb2-8eb5-6214d5078068\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:02.778473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.778413 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c0f995-78f8-4a88-9cef-7c976677ebdc-tls-certs\") pod \"model-serving-api-75bd89fdfc-59st9\" (UID: \"58c0f995-78f8-4a88-9cef-7c976677ebdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:02.778473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.778443 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw84p\" (UniqueName: \"kubernetes.io/projected/58c0f995-78f8-4a88-9cef-7c976677ebdc-kube-api-access-qw84p\") pod \"model-serving-api-75bd89fdfc-59st9\" (UID: \"58c0f995-78f8-4a88-9cef-7c976677ebdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:02.780679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.780657 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e141d889-3d52-4eb2-8eb5-6214d5078068-cert\") pod \"kserve-controller-manager-776cc66d47-sbh4k\" (UID: \"e141d889-3d52-4eb2-8eb5-6214d5078068\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:02.796563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.796540 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjtd\" (UniqueName: \"kubernetes.io/projected/e141d889-3d52-4eb2-8eb5-6214d5078068-kube-api-access-5tjtd\") pod \"kserve-controller-manager-776cc66d47-sbh4k\" (UID: \"e141d889-3d52-4eb2-8eb5-6214d5078068\") " pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:02.850404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.850330 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:02.879246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.879210 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c0f995-78f8-4a88-9cef-7c976677ebdc-tls-certs\") pod \"model-serving-api-75bd89fdfc-59st9\" (UID: \"58c0f995-78f8-4a88-9cef-7c976677ebdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:02.879403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.879262 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw84p\" (UniqueName: \"kubernetes.io/projected/58c0f995-78f8-4a88-9cef-7c976677ebdc-kube-api-access-qw84p\") pod \"model-serving-api-75bd89fdfc-59st9\" (UID: \"58c0f995-78f8-4a88-9cef-7c976677ebdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:02.881594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.881568 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c0f995-78f8-4a88-9cef-7c976677ebdc-tls-certs\") pod \"model-serving-api-75bd89fdfc-59st9\" (UID: \"58c0f995-78f8-4a88-9cef-7c976677ebdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:02.916808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.916778 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw84p\" (UniqueName: \"kubernetes.io/projected/58c0f995-78f8-4a88-9cef-7c976677ebdc-kube-api-access-qw84p\") pod \"model-serving-api-75bd89fdfc-59st9\" (UID: \"58c0f995-78f8-4a88-9cef-7c976677ebdc\") " pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:02.925028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:02.925003 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:03.024125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:03.024091 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:03.168380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:03.168354 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:03.218266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:03.218239 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-sbh4k"] Oct 30 18:05:03.222206 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:05:03.222181 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode141d889_3d52_4eb2_8eb5_6214d5078068.slice/crio-eb94e2c58bd9e9831fb172c0fee5cd71105947846c0af63938075c1df14c8a41 WatchSource:0}: Error finding container eb94e2c58bd9e9831fb172c0fee5cd71105947846c0af63938075c1df14c8a41: Status 404 returned error can't find the container with id eb94e2c58bd9e9831fb172c0fee5cd71105947846c0af63938075c1df14c8a41 Oct 30 18:05:03.316089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:03.316058 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-59st9"] Oct 30 18:05:03.320462 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:05:03.320438 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c0f995_78f8_4a88_9cef_7c976677ebdc.slice/crio-2083db21ed0597a0b4a09ba1370f29f026f6baaf69954c85a5a8d8339e2a170c WatchSource:0}: Error finding container 2083db21ed0597a0b4a09ba1370f29f026f6baaf69954c85a5a8d8339e2a170c: Status 404 returned error can't find the container with id 2083db21ed0597a0b4a09ba1370f29f026f6baaf69954c85a5a8d8339e2a170c Oct 30 18:05:04.056808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:04.056767 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-59st9" event={"ID":"58c0f995-78f8-4a88-9cef-7c976677ebdc","Type":"ContainerStarted","Data":"2083db21ed0597a0b4a09ba1370f29f026f6baaf69954c85a5a8d8339e2a170c"} Oct 30 18:05:04.058482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:04.058449 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" event={"ID":"e141d889-3d52-4eb2-8eb5-6214d5078068","Type":"ContainerStarted","Data":"eb94e2c58bd9e9831fb172c0fee5cd71105947846c0af63938075c1df14c8a41"} Oct 30 18:05:04.803898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:04.803861 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:04.904137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:04.904104 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:05.018779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:05.018743 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:05.123039 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:05.123008 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:05.231924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:05.231889 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:06.071129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:06.071088 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-59st9" event={"ID":"58c0f995-78f8-4a88-9cef-7c976677ebdc","Type":"ContainerStarted","Data":"c73e9f557c51f0bc3561e7faf5ee07ce26a7f64ffb14d573c6d4c7b2c3fb00f1"} Oct 30 18:05:06.071599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:06.071214 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:06.072870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:06.072843 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" event={"ID":"e141d889-3d52-4eb2-8eb5-6214d5078068","Type":"ContainerStarted","Data":"25f48813784c4b33b7a68e0158b113939ed0fd3297cdcb9b6641f6796f8cdf34"} Oct 30 18:05:06.072971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:06.072887 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:06.132917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:06.132873 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-59st9" podStartSLOduration=2.288192427 podStartE2EDuration="4.132856111s" totalImagesPullingTime="1.844663684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:06.131193478 +0000 UTC m=+402.955587769" watchObservedRunningTime="2026-10-30 18:05:06.132856111 +0000 UTC m=+402.957250402" Oct 30 18:05:06.167237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:06.167182 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" podStartSLOduration=1.8543507780000001 podStartE2EDuration="4.167169874s" totalImagesPullingTime="2.312819096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:06.16527264 +0000 UTC m=+402.989666952" watchObservedRunningTime="2026-10-30 18:05:06.167169874 +0000 UTC m=+402.991564165" Oct 30 18:05:06.223564 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:05:06.223519 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:05:08.820082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:08.820048 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:08.999367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:08.999334 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:12.161134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:12.161102 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:12.290664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:12.290626 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:17.081716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:17.081690 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-59st9" Oct 30 18:05:34.883907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:34.883874 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:35.005214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:35.005178 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:36.227657 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:05:36.227622 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:05:36.348174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:36.348141 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:36.504595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:36.504520 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:37.082739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:37.082710 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-sbh4k" Oct 30 18:05:47.302801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:47.302760 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:47.406097 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:05:47.406066 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:06.224389 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:06:06.224350 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:06:12.256382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:12.256348 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:35.734736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:35.734651 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:36.224658 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:06:36.224623 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:06:36.530848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:36.530747 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:45.438912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.438877 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:45.465569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.465537 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fm7rx"] Oct 30 18:06:45.469132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.469111 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:45.493606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.493582 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 18:06:45.493764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.493707 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-2cdql\"" Oct 30 18:06:45.493878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.493802 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 18:06:45.495383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.495367 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fm7rx"] Oct 30 18:06:45.534805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.534774 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:45.569690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.569663 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6dx\" (UniqueName: \"kubernetes.io/projected/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-kube-api-access-vt6dx\") pod \"ai-gateway-controller-7d85675f98-fm7rx\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:45.569811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.569699 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-fm7rx\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:45.670860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.670816 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6dx\" (UniqueName: \"kubernetes.io/projected/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-kube-api-access-vt6dx\") pod \"ai-gateway-controller-7d85675f98-fm7rx\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:45.670972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.670870 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-fm7rx\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:45.673362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.673336 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-fm7rx\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:45.689763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.689703 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6dx\" (UniqueName: \"kubernetes.io/projected/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-kube-api-access-vt6dx\") pod \"ai-gateway-controller-7d85675f98-fm7rx\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:45.781548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:45.781520 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:46.072023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:46.071953 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fm7rx"] Oct 30 18:06:46.076919 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:06:46.076895 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e196e35_a47d_4d49_9c3f_384f5f23bfe9.slice/crio-3f70b9b67724738f158af160c09e6f1a3c0bdebed4a7ac317442c9a07ace78cc WatchSource:0}: Error finding container 3f70b9b67724738f158af160c09e6f1a3c0bdebed4a7ac317442c9a07ace78cc: Status 404 returned error can't find the container with id 3f70b9b67724738f158af160c09e6f1a3c0bdebed4a7ac317442c9a07ace78cc Oct 30 18:06:46.517971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:46.517935 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" event={"ID":"5e196e35-a47d-4d49-9c3f-384f5f23bfe9","Type":"ContainerStarted","Data":"3f70b9b67724738f158af160c09e6f1a3c0bdebed4a7ac317442c9a07ace78cc"} Oct 30 18:06:48.532608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:48.532580 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" event={"ID":"5e196e35-a47d-4d49-9c3f-384f5f23bfe9","Type":"ContainerStarted","Data":"89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed"} Oct 30 18:06:48.532953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:48.532704 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:48.571182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:48.571134 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" podStartSLOduration=1.184078541 podStartE2EDuration="3.571122586s" totalImagesPullingTime="2.387044045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:48.570787379 +0000 UTC m=+505.395181670" watchObservedRunningTime="2026-10-30 18:06:48.571122586 +0000 UTC m=+505.395516878" Oct 30 18:06:50.615012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.614978 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:50.714143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.714110 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:50.767698 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.767664 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6c557"] Oct 30 18:06:50.771196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.771175 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:50.798265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.798241 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 18:06:50.798265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.798264 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rfllq\"" Oct 30 18:06:50.800158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.800138 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6c557"] Oct 30 18:06:50.907390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.907353 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-97rs5"] Oct 30 18:06:50.910955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.910936 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:50.921383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.921365 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d6333d-538f-47bd-9240-f13efd118817-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6c557\" (UID: \"f0d6333d-538f-47bd-9240-f13efd118817\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:50.921474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.921396 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnt9\" (UniqueName: \"kubernetes.io/projected/f0d6333d-538f-47bd-9240-f13efd118817-kube-api-access-njnt9\") pod \"payload-pre-processing-7b98bb678c-6c557\" (UID: \"f0d6333d-538f-47bd-9240-f13efd118817\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:50.928788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:50.928762 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-97rs5"] Oct 30 18:06:51.022432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.022397 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d6333d-538f-47bd-9240-f13efd118817-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6c557\" (UID: \"f0d6333d-538f-47bd-9240-f13efd118817\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:51.022616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.022443 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnt9\" (UniqueName: \"kubernetes.io/projected/f0d6333d-538f-47bd-9240-f13efd118817-kube-api-access-njnt9\") pod \"payload-pre-processing-7b98bb678c-6c557\" (UID: \"f0d6333d-538f-47bd-9240-f13efd118817\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:51.022616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.022570 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdl84\" (UniqueName: \"kubernetes.io/projected/70164f26-3f3a-42b1-be62-923b183ee563-kube-api-access-jdl84\") pod \"payload-processing-5f78687d5b-97rs5\" (UID: \"70164f26-3f3a-42b1-be62-923b183ee563\") " pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:51.022749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.022681 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70164f26-3f3a-42b1-be62-923b183ee563-plugins-config-volume\") pod \"payload-processing-5f78687d5b-97rs5\" (UID: \"70164f26-3f3a-42b1-be62-923b183ee563\") " pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:51.023224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.023201 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d6333d-538f-47bd-9240-f13efd118817-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6c557\" (UID: \"f0d6333d-538f-47bd-9240-f13efd118817\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:51.040836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.040797 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnt9\" (UniqueName: \"kubernetes.io/projected/f0d6333d-538f-47bd-9240-f13efd118817-kube-api-access-njnt9\") pod \"payload-pre-processing-7b98bb678c-6c557\" (UID: \"f0d6333d-538f-47bd-9240-f13efd118817\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:51.084297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.084272 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:51.123708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.123674 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70164f26-3f3a-42b1-be62-923b183ee563-plugins-config-volume\") pod \"payload-processing-5f78687d5b-97rs5\" (UID: \"70164f26-3f3a-42b1-be62-923b183ee563\") " pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:51.123885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.123779 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdl84\" (UniqueName: \"kubernetes.io/projected/70164f26-3f3a-42b1-be62-923b183ee563-kube-api-access-jdl84\") pod \"payload-processing-5f78687d5b-97rs5\" (UID: \"70164f26-3f3a-42b1-be62-923b183ee563\") " pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:51.124515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.124493 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70164f26-3f3a-42b1-be62-923b183ee563-plugins-config-volume\") pod \"payload-processing-5f78687d5b-97rs5\" (UID: \"70164f26-3f3a-42b1-be62-923b183ee563\") " pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:51.141725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.141695 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdl84\" (UniqueName: \"kubernetes.io/projected/70164f26-3f3a-42b1-be62-923b183ee563-kube-api-access-jdl84\") pod \"payload-processing-5f78687d5b-97rs5\" (UID: \"70164f26-3f3a-42b1-be62-923b183ee563\") " pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:51.223639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.223561 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:51.374886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.374849 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6c557"] Oct 30 18:06:51.380050 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:06:51.380024 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d6333d_538f_47bd_9240_f13efd118817.slice/crio-1b332fd2fb24c8befb089439bcb23f3cab1bf22dd4e5951e548fdae44167e185 WatchSource:0}: Error finding container 1b332fd2fb24c8befb089439bcb23f3cab1bf22dd4e5951e548fdae44167e185: Status 404 returned error can't find the container with id 1b332fd2fb24c8befb089439bcb23f3cab1bf22dd4e5951e548fdae44167e185 Oct 30 18:06:51.516240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.516166 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-97rs5"] Oct 30 18:06:51.521066 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:06:51.521040 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70164f26_3f3a_42b1_be62_923b183ee563.slice/crio-08dc76522101f1ce9bfec5fdc5fb6e55301663d6746523404af8eee3a8422df4 WatchSource:0}: Error finding container 08dc76522101f1ce9bfec5fdc5fb6e55301663d6746523404af8eee3a8422df4: Status 404 returned error can't find the container with id 08dc76522101f1ce9bfec5fdc5fb6e55301663d6746523404af8eee3a8422df4 Oct 30 18:06:51.546434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.546406 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" event={"ID":"70164f26-3f3a-42b1-be62-923b183ee563","Type":"ContainerStarted","Data":"08dc76522101f1ce9bfec5fdc5fb6e55301663d6746523404af8eee3a8422df4"} Oct 30 18:06:51.547531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:51.547508 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" event={"ID":"f0d6333d-538f-47bd-9240-f13efd118817","Type":"ContainerStarted","Data":"1b332fd2fb24c8befb089439bcb23f3cab1bf22dd4e5951e548fdae44167e185"} Oct 30 18:06:54.564911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:54.564877 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" event={"ID":"f0d6333d-538f-47bd-9240-f13efd118817","Type":"ContainerStarted","Data":"b1ae92fcb11e6d264594c7b5a5157c2c008af3d8e2466cca708b05556684e8db"} Oct 30 18:06:54.565354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:54.564980 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:06:54.566343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:54.566320 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" event={"ID":"70164f26-3f3a-42b1-be62-923b183ee563","Type":"ContainerStarted","Data":"375e762c92b10135ed4fc528c8d4e827c351e2d285ab578253215de660f0cd45"} Oct 30 18:06:54.566482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:54.566445 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:06:54.602276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:54.602226 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" podStartSLOduration=2.255268161 podStartE2EDuration="4.602212041s" totalImagesPullingTime="2.34694388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:54.601371423 +0000 UTC m=+511.425765715" watchObservedRunningTime="2026-10-30 18:06:54.602212041 +0000 UTC m=+511.426606334" Oct 30 18:06:54.637830 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:54.637786 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" podStartSLOduration=2.326409208 podStartE2EDuration="4.637773571s" totalImagesPullingTime="2.311364363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:54.636924399 +0000 UTC m=+511.461318690" watchObservedRunningTime="2026-10-30 18:06:54.637773571 +0000 UTC m=+511.462167859" Oct 30 18:06:59.539086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.539059 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:06:59.825375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.825296 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-b665f6876-csbcr"] Oct 30 18:06:59.829259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.829240 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:06:59.849770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.849744 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b665f6876-csbcr"] Oct 30 18:06:59.897390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.897350 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4-metrics-certs\") pod \"ai-gateway-controller-b665f6876-csbcr\" (UID: \"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4\") " pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:06:59.897551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.897466 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ltc\" (UniqueName: \"kubernetes.io/projected/a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4-kube-api-access-b7ltc\") pod \"ai-gateway-controller-b665f6876-csbcr\" (UID: \"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4\") " pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:06:59.997874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.997843 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ltc\" (UniqueName: \"kubernetes.io/projected/a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4-kube-api-access-b7ltc\") pod \"ai-gateway-controller-b665f6876-csbcr\" (UID: \"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4\") " pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:06:59.998015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:06:59.997936 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4-metrics-certs\") pod \"ai-gateway-controller-b665f6876-csbcr\" (UID: \"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4\") " pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:07:00.000247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:00.000221 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4-metrics-certs\") pod \"ai-gateway-controller-b665f6876-csbcr\" (UID: \"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4\") " pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:07:00.015466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:00.015442 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ltc\" (UniqueName: \"kubernetes.io/projected/a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4-kube-api-access-b7ltc\") pod \"ai-gateway-controller-b665f6876-csbcr\" (UID: \"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4\") " pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:07:00.141773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:00.141743 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:07:00.438268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:00.438190 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b665f6876-csbcr"] Oct 30 18:07:00.443778 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:07:00.443749 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d9c49e_a7f9_44c4_9c8a_cd0d77da88e4.slice/crio-1a3e4559976068d79f1088a3019760c0af124ce973284992a3b96e7cda01b402 WatchSource:0}: Error finding container 1a3e4559976068d79f1088a3019760c0af124ce973284992a3b96e7cda01b402: Status 404 returned error can't find the container with id 1a3e4559976068d79f1088a3019760c0af124ce973284992a3b96e7cda01b402 Oct 30 18:07:00.599282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:00.599244 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" event={"ID":"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4","Type":"ContainerStarted","Data":"1a3e4559976068d79f1088a3019760c0af124ce973284992a3b96e7cda01b402"} Oct 30 18:07:01.605065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:01.605027 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" event={"ID":"a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4","Type":"ContainerStarted","Data":"a95464de98e675dffca18de536cc3a06a280a57ca359c5151259b8981e9d1a33"} Oct 30 18:07:01.605536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:01.605116 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:07:01.641296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:01.641248 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" podStartSLOduration=2.35127819 podStartE2EDuration="2.641234583s" totalImagesPullingTime="289.956393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:01.639691711 +0000 UTC m=+518.464086002" watchObservedRunningTime="2026-10-30 18:07:01.641234583 +0000 UTC m=+518.465628874" Oct 30 18:07:04.426502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:04.426467 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:05.574321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:05.574290 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6c557" Oct 30 18:07:05.574748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:05.574352 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-97rs5" Oct 30 18:07:06.223770 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:07:06.223735 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:07:09.256990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:09.256959 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:12.352101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:12.352063 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:12.612726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:12.612651 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-b665f6876-csbcr" Oct 30 18:07:12.679019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:12.678980 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fm7rx"] Oct 30 18:07:12.679222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:12.679177 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" podUID="5e196e35-a47d-4d49-9c3f-384f5f23bfe9" containerName="manager" containerID="cri-o://89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed" gracePeriod=30 Oct 30 18:07:12.924850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:12.924814 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:07:13.006587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.006553 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-kube-api-access-vt6dx\" (UniqueName: \"kubernetes.io/projected/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-kube-api-access-vt6dx\") pod \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " Oct 30 18:07:13.006766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.006616 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-metrics-certs\") pod \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\" (UID: \"5e196e35-a47d-4d49-9c3f-384f5f23bfe9\") " Oct 30 18:07:13.008813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.008783 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-metrics-certs" pod "5e196e35-a47d-4d49-9c3f-384f5f23bfe9" (UID: "5e196e35-a47d-4d49-9c3f-384f5f23bfe9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:13.008813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.008801 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-kube-api-access-vt6dx" pod "5e196e35-a47d-4d49-9c3f-384f5f23bfe9" (UID: "5e196e35-a47d-4d49-9c3f-384f5f23bfe9"). InnerVolumeSpecName "kube-api-access-vt6dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:13.108060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.108022 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt6dx\" (UniqueName: \"kubernetes.io/projected/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-kube-api-access-vt6dx\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:07:13.108060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.108052 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e196e35-a47d-4d49-9c3f-384f5f23bfe9-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:07:13.661955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.661921 4582 generic.go:356] "Generic (PLEG): container finished" podID="5e196e35-a47d-4d49-9c3f-384f5f23bfe9" containerID="89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed" exitCode=0 Oct 30 18:07:13.662362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.661982 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" Oct 30 18:07:13.662362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.662014 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" event={"ID":"5e196e35-a47d-4d49-9c3f-384f5f23bfe9","Type":"ContainerDied","Data":"89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed"} Oct 30 18:07:13.662362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.662070 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fm7rx" event={"ID":"5e196e35-a47d-4d49-9c3f-384f5f23bfe9","Type":"ContainerDied","Data":"3f70b9b67724738f158af160c09e6f1a3c0bdebed4a7ac317442c9a07ace78cc"} Oct 30 18:07:13.662362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.662092 4582 scope.go:122] "RemoveContainer" containerID="89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed" Oct 30 18:07:13.670205 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.670190 4582 scope.go:122] "RemoveContainer" containerID="89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed" Oct 30 18:07:13.670462 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:07:13.670445 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed\": container with ID starting with 89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed not found: ID does not exist" containerID="89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed" Oct 30 18:07:13.670509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.670471 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed"} err="failed to get container status \"89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed\": rpc error: code = NotFound desc = could not find container \"89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed\": container with ID starting with 89af199b0e48e0c82718264badc2b99a3a0ef808f2a88d490444a7744db69fed not found: ID does not exist" Oct 30 18:07:13.707345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.707290 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fm7rx"] Oct 30 18:07:13.716737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:13.716716 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fm7rx"] Oct 30 18:07:14.353620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:14.353587 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:14.878188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:14.878160 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:15.298771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:15.298695 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e196e35-a47d-4d49-9c3f-384f5f23bfe9" path="/var/lib/kubelet/pods/5e196e35-a47d-4d49-9c3f-384f5f23bfe9/volumes" Oct 30 18:07:16.209894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:16.209861 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:16.343452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:16.343425 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:19.533963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.533928 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6"] Oct 30 18:07:19.534409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.534369 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5e196e35-a47d-4d49-9c3f-384f5f23bfe9" containerName="manager" Oct 30 18:07:19.534409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.534386 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e196e35-a47d-4d49-9c3f-384f5f23bfe9" containerName="manager" Oct 30 18:07:19.534545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.534525 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e196e35-a47d-4d49-9c3f-384f5f23bfe9" containerName="manager" Oct 30 18:07:19.537421 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.537400 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.549503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.549486 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:07:19.549600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.549571 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:07:19.558878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.558861 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 18:07:19.558966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.558896 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdx68\"" Oct 30 18:07:19.563675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.563656 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6"] Oct 30 18:07:19.594768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.594739 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:19.668539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.668513 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.668690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.668557 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.668690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.668602 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29f47\" (UniqueName: \"kubernetes.io/projected/8e5667fa-9807-46ac-80e6-ab902edfb512-kube-api-access-29f47\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.668690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.668653 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.668690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.668675 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.668847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.668778 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5667fa-9807-46ac-80e6-ab902edfb512-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770141 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770209 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29f47\" (UniqueName: \"kubernetes.io/projected/8e5667fa-9807-46ac-80e6-ab902edfb512-kube-api-access-29f47\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770239 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770254 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770294 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5667fa-9807-46ac-80e6-ab902edfb512-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770330 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770617 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770627 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.770756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.770705 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.772470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.772451 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e5667fa-9807-46ac-80e6-ab902edfb512-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.772758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.772738 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5667fa-9807-46ac-80e6-ab902edfb512-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.792919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.792859 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29f47\" (UniqueName: \"kubernetes.io/projected/8e5667fa-9807-46ac-80e6-ab902edfb512-kube-api-access-29f47\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6\" (UID: \"8e5667fa-9807-46ac-80e6-ab902edfb512\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:19.850896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:19.850868 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:20.143980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:20.143952 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6"] Oct 30 18:07:20.148312 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:07:20.148285 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5667fa_9807_46ac_80e6_ab902edfb512.slice/crio-8c68e18a1771db2d951284ab7e5af5f136085b79cdf56d08873b58f190bc782e WatchSource:0}: Error finding container 8c68e18a1771db2d951284ab7e5af5f136085b79cdf56d08873b58f190bc782e: Status 404 returned error can't find the container with id 8c68e18a1771db2d951284ab7e5af5f136085b79cdf56d08873b58f190bc782e Oct 30 18:07:20.699165 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:20.699127 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" event={"ID":"8e5667fa-9807-46ac-80e6-ab902edfb512","Type":"ContainerStarted","Data":"8c68e18a1771db2d951284ab7e5af5f136085b79cdf56d08873b58f190bc782e"} Oct 30 18:07:21.705386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:21.705352 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" event={"ID":"8e5667fa-9807-46ac-80e6-ab902edfb512","Type":"ContainerStarted","Data":"162218a7212213d1d279edd797379afe2a4de21ad53cb59d2e9aa1e9392305be"} Oct 30 18:07:21.741846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:21.741791 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" podStartSLOduration=1.741536542 podStartE2EDuration="2.741777627s" totalImagesPullingTime="1.000241085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:21.740686194 +0000 UTC m=+538.565080499" watchObservedRunningTime="2026-10-30 18:07:21.741777627 +0000 UTC m=+538.566171919" Oct 30 18:07:21.851934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:21.851896 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:23.014048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:23.013983 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:24.848092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:24.848031 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:26.621936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.621888 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh"] Oct 30 18:07:26.625861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.625833 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.650025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.649992 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 18:07:26.652412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.652388 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh"] Oct 30 18:07:26.682190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.682168 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:26.729410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.729391 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.729518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.729421 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0f4a581-7366-46ab-9f04-eebb2f9ca897-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.729518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.729445 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.729599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.729518 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.729599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.729574 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpfxl\" (UniqueName: \"kubernetes.io/projected/d0f4a581-7366-46ab-9f04-eebb2f9ca897-kube-api-access-gpfxl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.729695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.729605 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.830841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.830774 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.830841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.830834 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0f4a581-7366-46ab-9f04-eebb2f9ca897-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.831054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.830947 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.831054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.830985 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.831054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.831019 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfxl\" (UniqueName: \"kubernetes.io/projected/d0f4a581-7366-46ab-9f04-eebb2f9ca897-kube-api-access-gpfxl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.831054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.831047 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.831276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.831246 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.831325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.831282 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.831414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.831386 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.833228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.833204 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0f4a581-7366-46ab-9f04-eebb2f9ca897-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.833463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.833446 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0f4a581-7366-46ab-9f04-eebb2f9ca897-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.862448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.862424 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfxl\" (UniqueName: \"kubernetes.io/projected/d0f4a581-7366-46ab-9f04-eebb2f9ca897-kube-api-access-gpfxl\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nscmh\" (UID: \"d0f4a581-7366-46ab-9f04-eebb2f9ca897\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:26.941267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:26.941203 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:27.237218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:27.237124 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh"] Oct 30 18:07:27.243625 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:07:27.243597 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f4a581_7366_46ab_9f04_eebb2f9ca897.slice/crio-1410d6e3549bb2e7f860afe78694b313364a5b3c05fe6aa5b25f13202ab158a0 WatchSource:0}: Error finding container 1410d6e3549bb2e7f860afe78694b313364a5b3c05fe6aa5b25f13202ab158a0: Status 404 returned error can't find the container with id 1410d6e3549bb2e7f860afe78694b313364a5b3c05fe6aa5b25f13202ab158a0 Oct 30 18:07:27.734336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:27.734303 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" event={"ID":"d0f4a581-7366-46ab-9f04-eebb2f9ca897","Type":"ContainerStarted","Data":"1a15e62a6d8651a67f0e551c98e3d3ebb4a97d1ac5aa1ccea1efa1af5a13cfc8"} Oct 30 18:07:27.734722 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:27.734397 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" event={"ID":"d0f4a581-7366-46ab-9f04-eebb2f9ca897","Type":"ContainerStarted","Data":"1410d6e3549bb2e7f860afe78694b313364a5b3c05fe6aa5b25f13202ab158a0"} Oct 30 18:07:27.770690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:27.770645 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" podStartSLOduration=1.581878095 podStartE2EDuration="1.770632946s" totalImagesPullingTime="188.754851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:27.77000895 +0000 UTC m=+544.594403243" watchObservedRunningTime="2026-10-30 18:07:27.770632946 +0000 UTC m=+544.595027237" Oct 30 18:07:27.942250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:27.942220 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:28.209262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.209232 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5"] Oct 30 18:07:28.211969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.211948 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.223680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.223654 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 18:07:28.235619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.235591 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5"] Oct 30 18:07:28.265853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.265813 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:28.346657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.346626 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.346876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.346663 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.346876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.346792 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.346978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.346923 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcscc\" (UniqueName: \"kubernetes.io/projected/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-kube-api-access-dcscc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.346978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.346965 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.347070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.346990 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.447950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.447897 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.447961 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.447995 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.448017 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.448174 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.448256 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcscc\" (UniqueName: \"kubernetes.io/projected/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-kube-api-access-dcscc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.448551 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.448586 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.448791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.448746 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.450424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.450401 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.450952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.450931 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.465327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.465278 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcscc\" (UniqueName: \"kubernetes.io/projected/ed7bf2c0-385c-4a64-99fb-0913b3463bfa-kube-api-access-dcscc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5\" (UID: \"ed7bf2c0-385c-4a64-99fb-0913b3463bfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.525147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.525063 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:28.815415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:28.815333 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5"] Oct 30 18:07:28.820109 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:07:28.820080 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7bf2c0_385c_4a64_99fb_0913b3463bfa.slice/crio-474a4f7fbb5abb6faf71962053ccb8ab3f19fdaa72a1691b079c1ba0d3af926d WatchSource:0}: Error finding container 474a4f7fbb5abb6faf71962053ccb8ab3f19fdaa72a1691b079c1ba0d3af926d: Status 404 returned error can't find the container with id 474a4f7fbb5abb6faf71962053ccb8ab3f19fdaa72a1691b079c1ba0d3af926d Oct 30 18:07:29.745433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:29.745399 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" event={"ID":"ed7bf2c0-385c-4a64-99fb-0913b3463bfa","Type":"ContainerStarted","Data":"8d3f4f4b4bd236a105c5df082c6f9642e432714ee243f684ef3919014d53f8c2"} Oct 30 18:07:29.745623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:29.745462 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" event={"ID":"ed7bf2c0-385c-4a64-99fb-0913b3463bfa","Type":"ContainerStarted","Data":"474a4f7fbb5abb6faf71962053ccb8ab3f19fdaa72a1691b079c1ba0d3af926d"} Oct 30 18:07:29.788643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:29.788585 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" podStartSLOduration=1.611294717 podStartE2EDuration="1.788571305s" totalImagesPullingTime="177.276588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:29.785190889 +0000 UTC m=+546.609585181" watchObservedRunningTime="2026-10-30 18:07:29.788571305 +0000 UTC m=+546.612965594" Oct 30 18:07:29.851250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:29.851211 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:29.863674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:29.863651 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:30.339272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:30.339245 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:30.525906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:30.525879 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:30.761375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:30.761343 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6" Oct 30 18:07:33.711841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.711794 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj"] Oct 30 18:07:33.715672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.715650 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.728646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.728624 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 18:07:33.744988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.744960 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj"] Oct 30 18:07:33.796110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.796080 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.796266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.796121 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad83681-b01e-416a-bd8a-fafae54c4f56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.796266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.796141 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.796266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.796230 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l6pp\" (UniqueName: \"kubernetes.io/projected/bad83681-b01e-416a-bd8a-fafae54c4f56-kube-api-access-4l6pp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.796438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.796297 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.796493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.796437 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.803158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.803134 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:33.897875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.897815 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.897932 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.897970 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad83681-b01e-416a-bd8a-fafae54c4f56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.897998 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.898036 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l6pp\" (UniqueName: \"kubernetes.io/projected/bad83681-b01e-416a-bd8a-fafae54c4f56-kube-api-access-4l6pp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.898082 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.898290 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.898391 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.898455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.898434 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.900048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.900023 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bad83681-b01e-416a-bd8a-fafae54c4f56-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.900479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.900457 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bad83681-b01e-416a-bd8a-fafae54c4f56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:33.929966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:33.929939 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l6pp\" (UniqueName: \"kubernetes.io/projected/bad83681-b01e-416a-bd8a-fafae54c4f56-kube-api-access-4l6pp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj\" (UID: \"bad83681-b01e-416a-bd8a-fafae54c4f56\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:34.030846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:34.030745 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:34.331171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:34.331093 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj"] Oct 30 18:07:34.335463 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:07:34.335436 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad83681_b01e_416a_bd8a_fafae54c4f56.slice/crio-a4a84a7a184a711619349bec3c1e734e3151b5fa0ca7c623820e906998dc100a WatchSource:0}: Error finding container a4a84a7a184a711619349bec3c1e734e3151b5fa0ca7c623820e906998dc100a: Status 404 returned error can't find the container with id a4a84a7a184a711619349bec3c1e734e3151b5fa0ca7c623820e906998dc100a Oct 30 18:07:34.523150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:34.523120 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:34.631980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:34.631950 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:34.771179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:34.771144 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" event={"ID":"bad83681-b01e-416a-bd8a-fafae54c4f56","Type":"ContainerStarted","Data":"9080d40c3a3325e2eaf1e3addf0a3375c7ce4f9cc0480382de7f634ad8e7b079"} Oct 30 18:07:34.771604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:34.771195 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" event={"ID":"bad83681-b01e-416a-bd8a-fafae54c4f56","Type":"ContainerStarted","Data":"a4a84a7a184a711619349bec3c1e734e3151b5fa0ca7c623820e906998dc100a"} Oct 30 18:07:34.811319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:34.811268 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" podStartSLOduration=1.611141195 podStartE2EDuration="1.811253824s" totalImagesPullingTime="200.112629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:34.809780197 +0000 UTC m=+551.634174488" watchObservedRunningTime="2026-10-30 18:07:34.811253824 +0000 UTC m=+551.635648165" Oct 30 18:07:35.030884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:35.030848 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:35.931934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:35.931904 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:36.224131 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:07:36.224049 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:07:36.941629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:36.941597 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:36.954191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:36.954172 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:37.795620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:37.795592 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nscmh" Oct 30 18:07:38.526346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:38.526308 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:38.538974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:38.538949 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:38.799730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:38.799650 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5" Oct 30 18:07:44.031926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:44.031894 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:44.044694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:44.044668 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:44.827334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:44.827306 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" Oct 30 18:07:49.956540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:49.956460 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7"] Oct 30 18:07:49.960021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:49.960000 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:49.982762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:49.982730 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:07:49.982762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:49.982745 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xfzjq\"" Oct 30 18:07:49.988675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:49.988647 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7"] Oct 30 18:07:50.146048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146012 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146069 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146173 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pf9\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-kube-api-access-f2pf9\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146205 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146227 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146298 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146341 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146390 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.146453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.146425 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247599 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247633 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247663 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247686 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247725 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247767 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pf9\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-kube-api-access-f2pf9\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247801 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247906 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.247947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.247935 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.248237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.248213 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.248328 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.248299 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.248437 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.248417 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.248492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.248424 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.248492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.248475 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.249891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.249873 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.250323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.250307 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.268979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.268956 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.269080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.269020 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pf9\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-kube-api-access-f2pf9\") pod \"maas-default-gateway-openshift-default-76597d6855-w98z7\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.273550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.273530 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:50.579121 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.579037 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7"] Oct 30 18:07:50.584137 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:07:50.584108 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c11a1e_43a9_46ff_ba67_72fe77cc0508.slice/crio-d23df189b048a7b5e983ad94e514de7dc84f0991bb5c730694a4112bd138d30c WatchSource:0}: Error finding container d23df189b048a7b5e983ad94e514de7dc84f0991bb5c730694a4112bd138d30c: Status 404 returned error can't find the container with id d23df189b048a7b5e983ad94e514de7dc84f0991bb5c730694a4112bd138d30c Oct 30 18:07:50.846720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:50.846637 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" event={"ID":"b7c11a1e-43a9-46ff-ba67-72fe77cc0508","Type":"ContainerStarted","Data":"d23df189b048a7b5e983ad94e514de7dc84f0991bb5c730694a4112bd138d30c"} Oct 30 18:07:52.433573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:52.433543 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:52.535415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:52.535378 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:53.222935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:53.222906 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:07:53.223010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:53.222973 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:07:53.223010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:53.223001 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:07:53.872453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:53.872413 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" event={"ID":"b7c11a1e-43a9-46ff-ba67-72fe77cc0508","Type":"ContainerStarted","Data":"dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa"} Oct 30 18:07:53.928629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:53.928573 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podStartSLOduration=2.292689763 podStartE2EDuration="4.928560099s" totalImagesPullingTime="2.635870336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:53.917915721 +0000 UTC m=+570.742310032" watchObservedRunningTime="2026-10-30 18:07:53.928560099 +0000 UTC m=+570.752954391" Oct 30 18:07:54.274153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:54.274072 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:07:54.275940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:54.275911 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w98z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 18:07:54.276086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:54.275962 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 18:07:55.274917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:55.274884 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w98z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 18:07:55.275301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:55.274936 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 18:07:56.274048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:56.274007 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w98z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 18:07:56.274262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:56.274081 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 18:07:57.274644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:57.274608 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w98z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 18:07:57.275048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:57.274666 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 18:07:58.275516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:58.275482 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w98z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 18:07:58.275955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:58.275536 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 18:07:59.274067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:59.274031 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w98z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 18:07:59.274211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:07:59.274097 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 18:08:00.274520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:00.274488 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w98z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 18:08:00.274887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:00.274539 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 18:08:01.427356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:01.427315 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:08:01.428061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:01.428043 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:08:01.428740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:01.428716 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:08:04.011307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:04.011277 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:04.119961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:04.119928 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:05.014154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.014123 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq"] Oct 30 18:08:05.020946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.020901 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.048453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.048425 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq"] Oct 30 18:08:05.051655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.051631 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p"] Oct 30 18:08:05.055547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.055529 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.080644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080604 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.080782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080651 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.080782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080677 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.080782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080717 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.080782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080768 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7km\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-kube-api-access-nz7km\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.081027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080812 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.081027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080865 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.081027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080889 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.081027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080932 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.081027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080956 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.081027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.080990 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.081027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.081018 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.081302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.081045 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.081302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.081089 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.081302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.081114 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.081302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.081136 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a90fe9e-b661-483d-a375-d5bfe196be63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.081422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.081316 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfg5\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-kube-api-access-kxfg5\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.081422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.081381 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.085161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.085139 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p"] Oct 30 18:08:05.182368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182332 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182372 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182412 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.182609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182433 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182462 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182488 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.182609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182513 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182638 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182683 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182711 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a90fe9e-b661-483d-a375-d5bfe196be63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182774 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfg5\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-kube-api-access-kxfg5\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182787 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182807 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182809 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182857 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.182918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182905 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182948 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182954 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.182974 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183027 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183066 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183082 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7km\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-kube-api-access-nz7km\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183131 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183142 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183178 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183286 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.183389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183344 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.184055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.183569 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a90fe9e-b661-483d-a375-d5bfe196be63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.185428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.185401 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.185554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.185511 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.185554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.185539 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.185674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.185580 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.199470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.199435 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.199640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.199555 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfg5\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-kube-api-access-kxfg5\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.200119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.200099 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xkvhq\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.200219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.200123 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7km\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-kube-api-access-nz7km\") pod \"maas-default-gateway-openshift-default-76597d6855-v7b8p\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.334749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.334679 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:05.376704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.376673 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:05.641125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.641057 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq"] Oct 30 18:08:05.645151 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:08:05.645120 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc25d92_99d6_40db_9d7f_c2382e18d23b.slice/crio-479b4080e774ab240fba64a55fc1d7f3f148c17961701ed65957dadd8d813e9c WatchSource:0}: Error finding container 479b4080e774ab240fba64a55fc1d7f3f148c17961701ed65957dadd8d813e9c: Status 404 returned error can't find the container with id 479b4080e774ab240fba64a55fc1d7f3f148c17961701ed65957dadd8d813e9c Oct 30 18:08:05.647652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.647622 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:08:05.647737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.647688 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:08:05.647737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.647719 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:08:05.689967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.689942 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p"] Oct 30 18:08:05.694625 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:08:05.694592 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a90fe9e_b661_483d_a375_d5bfe196be63.slice/crio-6ae918540135a19efe4cb8861ae76e33466e818b3dc707ee42c5efc426e8ef47 WatchSource:0}: Error finding container 6ae918540135a19efe4cb8861ae76e33466e818b3dc707ee42c5efc426e8ef47: Status 404 returned error can't find the container with id 6ae918540135a19efe4cb8861ae76e33466e818b3dc707ee42c5efc426e8ef47 Oct 30 18:08:05.696942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.696912 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:08:05.697022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.696974 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:08:05.697022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.697000 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:08:05.930138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.930041 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" event={"ID":"1fc25d92-99d6-40db-9d7f-c2382e18d23b","Type":"ContainerStarted","Data":"804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18"} Oct 30 18:08:05.930138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.930106 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" event={"ID":"1fc25d92-99d6-40db-9d7f-c2382e18d23b","Type":"ContainerStarted","Data":"479b4080e774ab240fba64a55fc1d7f3f148c17961701ed65957dadd8d813e9c"} Oct 30 18:08:05.931882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.931809 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" event={"ID":"6a90fe9e-b661-483d-a375-d5bfe196be63","Type":"ContainerStarted","Data":"cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb"} Oct 30 18:08:05.932034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.931899 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" event={"ID":"6a90fe9e-b661-483d-a375-d5bfe196be63","Type":"ContainerStarted","Data":"6ae918540135a19efe4cb8861ae76e33466e818b3dc707ee42c5efc426e8ef47"} Oct 30 18:08:05.998303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:05.998228 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podStartSLOduration=1.99820961 podStartE2EDuration="1.99820961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:05.998050006 +0000 UTC m=+582.822444325" watchObservedRunningTime="2026-10-30 18:08:05.99820961 +0000 UTC m=+582.822603903" Oct 30 18:08:06.048194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:06.048143 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podStartSLOduration=1.0481291640000001 podStartE2EDuration="1.048129164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:06.047556177 +0000 UTC m=+582.871950468" watchObservedRunningTime="2026-10-30 18:08:06.048129164 +0000 UTC m=+582.872523456" Oct 30 18:08:06.230254 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:08:06.230178 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:08:06.335870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:06.335768 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:06.337447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:06.337410 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.337566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:06.337470 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:08:06.379244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:06.379200 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:06.392112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:06.391940 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.392112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:06.392008 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:08:07.335236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:07.335198 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.335648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:07.335254 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:08:07.381844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:07.381365 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.381844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:07.381443 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:08:08.335545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:08.335514 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:08.335952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:08.335567 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:08:08.377296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:08.377260 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:08.377568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:08.377543 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:08:09.336009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:09.335975 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:09.336390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:09.336024 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:08:09.392596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:09.392560 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:09.392792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:09.392632 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:08:10.335885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:10.335843 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:10.336067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:10.335904 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:08:10.388844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:10.387918 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:10.388844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:10.387991 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:08:11.335463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:11.335435 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.335694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:11.335674 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:08:11.380249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:11.380211 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.380839 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:11.380786 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:08:12.335325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:12.335289 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xkvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:12.335540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:12.335343 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:08:12.377555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:12.377512 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:12.377719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:12.377576 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:08:12.458643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:12.458591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:13.981379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:13.981345 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:13.982118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:13.982097 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:13.983179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:13.983156 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:08:14.007628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:14.007599 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:14.007869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:14.007850 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:14.008654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:14.008634 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:08:33.297288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:33.297253 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:33.386581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:08:33.386553 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:36.224255 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:08:36.224218 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:09:06.223526 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:09:06.223491 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:09:10.711478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:10.711442 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:10.828764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:10.828732 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.132856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.132814 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.200215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.200182 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw"] Oct 30 18:09:11.204064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.204041 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.230871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.230845 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.231021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.230983 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-594f46d3-openshift-default-dockercfg-kzgf6\"" Oct 30 18:09:11.234202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.234175 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw"] Oct 30 18:09:11.261305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261279 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-data\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261337 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqlh\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-kube-api-access-ldqlh\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261419 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-podinfo\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261457 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-socket\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261493 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-certs\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261583 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-envoy\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261649 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-credential-socket\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261682 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b64be5-1cd8-46f8-989e-a79428dbae84-istiod-ca-cert\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.261804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.261789 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-token\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.315270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.315236 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5"] Oct 30 18:09:11.319576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.319554 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.331392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.331371 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-54599e13-openshift-default-dockercfg-hvhfb\"" Oct 30 18:09:11.338581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.338538 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5"] Oct 30 18:09:11.362836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.362785 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4fm\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-kube-api-access-gf4fm\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.362977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.362853 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-credential-socket\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.362977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.362921 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b64be5-1cd8-46f8-989e-a79428dbae84-istiod-ca-cert\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.362977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.362947 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-podinfo\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.362977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.362975 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-credential-socket\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.363215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.362994 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-certs\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.363215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363013 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-token\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.363215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363137 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istiod-ca-cert\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.363215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363190 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-token\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363280 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-credential-socket\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363297 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-socket\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.363426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363349 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-envoy\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.363426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363405 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-data\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363456 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqlh\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-kube-api-access-ldqlh\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363498 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-data\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.363640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363529 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-podinfo\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363613 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b64be5-1cd8-46f8-989e-a79428dbae84-istiod-ca-cert\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363677 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-socket\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363717 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-certs\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363763 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-envoy\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.363891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.363793 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-data\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.365047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.364999 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-certs\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.365751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.365723 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.366938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.366919 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-podinfo\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.368087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.368063 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-socket\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.370311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.370283 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-envoy\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.379971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.379946 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-token\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.380089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.380072 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqlh\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-kube-api-access-ldqlh\") pod \"e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.464636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464554 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-certs\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.464636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464601 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-token\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.464874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464647 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istiod-ca-cert\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.464874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464702 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-socket\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.464874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464727 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-envoy\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.464874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464800 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-data\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464929 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4fm\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-kube-api-access-gf4fm\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.464961 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-podinfo\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.465001 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-credential-socket\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.465016 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-certs\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.465097 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-socket\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.465275 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-data\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.465468 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istiod-ca-cert\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.465575 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-credential-socket\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.465804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.465787 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.467223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.467197 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-envoy\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.467528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.467511 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-podinfo\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.481471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.481447 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4fm\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-kube-api-access-gf4fm\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.481557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.481479 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-token\") pod \"e2e-ait-54599e13-openshift-default-d8998f799-mwtl5\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.517458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.517441 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:11.604043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.584872 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.636384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.636348 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:11.705804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.705773 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.829532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.829445 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.865308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.865273 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw"] Oct 30 18:09:11.871884 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:09:11.871848 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b64be5_1cd8_46f8_989e_a79428dbae84.slice/crio-5ea3ea8a9f6655ac9f1540de27919214893404cb9b69d45504b8f01ff670020a WatchSource:0}: Error finding container 5ea3ea8a9f6655ac9f1540de27919214893404cb9b69d45504b8f01ff670020a: Status 404 returned error can't find the container with id 5ea3ea8a9f6655ac9f1540de27919214893404cb9b69d45504b8f01ff670020a Oct 30 18:09:11.874442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.874404 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:11.874524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.874487 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:11.874566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.874527 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:11.977020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.976983 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5"] Oct 30 18:09:11.979513 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:09:11.979485 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bae5601_08f4_4f35_8b25_e8eda5a987b1.slice/crio-3f022556aff7ae8fbec1e17d86903d6e34809a7c0149faacac22972dd5da8e84 WatchSource:0}: Error finding container 3f022556aff7ae8fbec1e17d86903d6e34809a7c0149faacac22972dd5da8e84: Status 404 returned error can't find the container with id 3f022556aff7ae8fbec1e17d86903d6e34809a7c0149faacac22972dd5da8e84 Oct 30 18:09:11.981886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.981851 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:11.982032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.981930 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:11.982032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:11.981978 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:12.306167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.306136 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" event={"ID":"36b64be5-1cd8-46f8-989e-a79428dbae84","Type":"ContainerStarted","Data":"a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235"} Oct 30 18:09:12.306350 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.306197 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" event={"ID":"36b64be5-1cd8-46f8-989e-a79428dbae84","Type":"ContainerStarted","Data":"5ea3ea8a9f6655ac9f1540de27919214893404cb9b69d45504b8f01ff670020a"} Oct 30 18:09:12.307707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.307678 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" event={"ID":"0bae5601-08f4-4f35-8b25-e8eda5a987b1","Type":"ContainerStarted","Data":"1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc"} Oct 30 18:09:12.307858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.307716 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" event={"ID":"0bae5601-08f4-4f35-8b25-e8eda5a987b1","Type":"ContainerStarted","Data":"3f022556aff7ae8fbec1e17d86903d6e34809a7c0149faacac22972dd5da8e84"} Oct 30 18:09:12.343369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.343319 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podStartSLOduration=1.343301503 podStartE2EDuration="1.343301503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:12.341960447 +0000 UTC m=+649.166354738" watchObservedRunningTime="2026-10-30 18:09:12.343301503 +0000 UTC m=+649.167695798" Oct 30 18:09:12.379608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.379560 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podStartSLOduration=1.3795476770000001 podStartE2EDuration="1.379547677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:12.377457928 +0000 UTC m=+649.201852250" watchObservedRunningTime="2026-10-30 18:09:12.379547677 +0000 UTC m=+649.203941969" Oct 30 18:09:12.517707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.517673 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:12.522460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.522437 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:12.534800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.534777 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:12.616434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.616338 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:12.637349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.637327 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:12.990323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.990290 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x"] Oct 30 18:09:12.995000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:12.994967 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.007009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.006977 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a521cf49-openshift-default-dockercfg-qltms\"" Oct 30 18:09:13.014500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.014469 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x"] Oct 30 18:09:13.035265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.035235 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:13.127986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.127882 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd594cec-adc6-48c7-aad6-45c444ce6c59-istiod-ca-cert\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.127986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.127954 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-data\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.128256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.128077 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-socket\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.128256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.128139 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-credential-socket\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.128256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.128190 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-envoy\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.128256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.128221 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-token\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.128256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.128249 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfc9l\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-kube-api-access-mfc9l\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.128538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.128286 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-certs\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.128538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.128340 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-podinfo\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.170791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.170759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:13.229709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.229677 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-envoy\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.229709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.229712 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-token\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.229730 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfc9l\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-kube-api-access-mfc9l\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.229755 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-certs\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.229774 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-podinfo\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.229863 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd594cec-adc6-48c7-aad6-45c444ce6c59-istiod-ca-cert\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.229929 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-data\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.230061 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-socket\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.230123 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-credential-socket\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.230281 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-certs\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.230308 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-data\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.230445 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-socket\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.230491 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-credential-socket\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.230713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.230689 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd594cec-adc6-48c7-aad6-45c444ce6c59-istiod-ca-cert\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.232079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.232063 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-envoy\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.232325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.232307 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-podinfo\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.257358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.257301 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfc9l\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-kube-api-access-mfc9l\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.257358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.257307 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-token\") pod \"e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.310132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.310106 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:13.312607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.312591 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:13.313450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.313434 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:13.623747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.623719 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x"] Oct 30 18:09:13.627500 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:09:13.627469 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd594cec_adc6_48c7_aad6_45c444ce6c59.slice/crio-827c3b9e7df9efa53239c3e9f94b766d7c784122db3f39e89c702adecbb364be WatchSource:0}: Error finding container 827c3b9e7df9efa53239c3e9f94b766d7c784122db3f39e89c702adecbb364be: Status 404 returned error can't find the container with id 827c3b9e7df9efa53239c3e9f94b766d7c784122db3f39e89c702adecbb364be Oct 30 18:09:13.629910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.629882 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:13.629977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.629943 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:13.629977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.629970 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:13.642869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.642851 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:13.944533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:13.944449 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:14.044538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.044511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:14.145358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.145323 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:14.279512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.279438 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:14.318728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.318694 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" event={"ID":"cd594cec-adc6-48c7-aad6-45c444ce6c59","Type":"ContainerStarted","Data":"469d3fae1330268bc49b420546b608c8c6aca797ff8b5fea28a04f0047093122"} Oct 30 18:09:14.318893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.318746 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" event={"ID":"cd594cec-adc6-48c7-aad6-45c444ce6c59","Type":"ContainerStarted","Data":"827c3b9e7df9efa53239c3e9f94b766d7c784122db3f39e89c702adecbb364be"} Oct 30 18:09:14.319152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.319136 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:14.320017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.320000 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:14.369493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.369447 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podStartSLOduration=2.369433449 podStartE2EDuration="2.369433449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:14.358615701 +0000 UTC m=+651.183010004" watchObservedRunningTime="2026-10-30 18:09:14.369433449 +0000 UTC m=+651.193827815" Oct 30 18:09:14.767962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.767926 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:14.869555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:14.869526 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:15.311245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.311216 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:15.317584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.317558 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:15.324318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.324290 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:15.324795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.324777 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:15.542712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.542668 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:15.640870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.640816 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:15.743001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.742963 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:15.847015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.846979 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:15.967850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:15.967742 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:16.927387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:16.927357 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.037090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:17.037059 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.450093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:17.450061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.555365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:17.555331 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.649736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:17.649704 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.754078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:17.753987 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.855094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:17.855060 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:18.399139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:18.399107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:18.497175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:18.497148 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:19.274690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:19.274663 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:19.373808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:19.373777 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:19.492659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:19.492624 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:20.553334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:20.553298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:20.654051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:20.654022 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:21.205929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:21.205887 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:21.301716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:21.301678 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:21.401594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:21.401560 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:23.046719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:23.046687 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:24.819071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:24.819030 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:24.919010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:24.918969 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:26.570846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:26.566314 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:27.363845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:27.360636 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:27.463845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:27.463277 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:27.702857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:27.702811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:27.800625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:27.800594 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:28.152846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.147961 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:28.232841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.229685 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9"] Oct 30 18:09:28.233953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.233932 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.254850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.253180 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ba56e909-openshift-default-dockercfg-86fhn\"" Oct 30 18:09:28.306954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.306912 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9"] Oct 30 18:09:28.309091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.309044 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:28.324307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324281 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-credential-socket\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324315 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-data\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324343 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9d6t\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-kube-api-access-m9d6t\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324379 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-podinfo\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324406 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-socket\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324426 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istiod-ca-cert\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324465 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-envoy\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324498 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-token\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.324594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.324518 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-certs\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.425740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.425649 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-certs\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.425740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.425720 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-credential-socket\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.425752 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-data\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.425788 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9d6t\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-kube-api-access-m9d6t\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.425876 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-podinfo\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.425922 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-socket\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.425951 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istiod-ca-cert\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.426019 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-envoy\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.426091 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-token\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.426114 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-certs\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.426125 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-data\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.426314 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-socket\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.426714 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-credential-socket\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.426896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.426714 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istiod-ca-cert\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.428534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.428519 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-envoy\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.428639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.428593 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-podinfo\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.429281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.429262 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:28.451906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.442052 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-token\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.452057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.442076 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9d6t\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-kube-api-access-m9d6t\") pod \"e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.612849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.610380 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:28.994493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.993900 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9"] Oct 30 18:09:28.995315 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:09:28.995286 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb2e704_471b_4172_98e8_10ec56b4f5a6.slice/crio-7fd97bd0732b6fb1eacb3c6d7bfd6e32017caa4816dd94a74a8cf7dfe9f8e797 WatchSource:0}: Error finding container 7fd97bd0732b6fb1eacb3c6d7bfd6e32017caa4816dd94a74a8cf7dfe9f8e797: Status 404 returned error can't find the container with id 7fd97bd0732b6fb1eacb3c6d7bfd6e32017caa4816dd94a74a8cf7dfe9f8e797 Oct 30 18:09:28.997877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.997843 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:28.997958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.997933 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:28.997994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:28.997958 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:29.426292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:29.426252 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" event={"ID":"dcb2e704-471b-4172-98e8-10ec56b4f5a6","Type":"ContainerStarted","Data":"55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9"} Oct 30 18:09:29.426455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:29.426308 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" event={"ID":"dcb2e704-471b-4172-98e8-10ec56b4f5a6","Type":"ContainerStarted","Data":"7fd97bd0732b6fb1eacb3c6d7bfd6e32017caa4816dd94a74a8cf7dfe9f8e797"} Oct 30 18:09:29.482159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:29.482087 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" podStartSLOduration=1.482065241 podStartE2EDuration="1.482065241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:29.463966736 +0000 UTC m=+666.288361048" watchObservedRunningTime="2026-10-30 18:09:29.482065241 +0000 UTC m=+666.306459534" Oct 30 18:09:29.610925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:29.610892 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:29.947924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:29.947889 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:30.615907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:30.615850 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:31.021080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:31.021041 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:31.157745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:31.157710 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:31.436171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:31.436142 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:31.436948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:31.436928 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:31.903147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:31.903082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:33.579677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:33.579644 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:33.686768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:33.686709 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:33.782465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:33.782428 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:34.455633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.455553 4582 generic.go:356] "Generic (PLEG): container finished" podID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerID="1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc" exitCode=0 Oct 30 18:09:34.455845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.455633 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" event={"ID":"0bae5601-08f4-4f35-8b25-e8eda5a987b1","Type":"ContainerDied","Data":"1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc"} Oct 30 18:09:34.456156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.456138 4582 scope.go:122] "RemoveContainer" containerID="1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc" Oct 30 18:09:34.457292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.457269 4582 generic.go:356] "Generic (PLEG): container finished" podID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerID="469d3fae1330268bc49b420546b608c8c6aca797ff8b5fea28a04f0047093122" exitCode=0 Oct 30 18:09:34.457391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.457342 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" event={"ID":"cd594cec-adc6-48c7-aad6-45c444ce6c59","Type":"ContainerDied","Data":"469d3fae1330268bc49b420546b608c8c6aca797ff8b5fea28a04f0047093122"} Oct 30 18:09:34.457545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.457516 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:34.457621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.457602 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:34.457675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.457655 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:34.457758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.457743 4582 scope.go:122] "RemoveContainer" containerID="469d3fae1330268bc49b420546b608c8c6aca797ff8b5fea28a04f0047093122" Oct 30 18:09:34.458925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.458904 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:34.458969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.458956 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:34.459013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.458988 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:34.918637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.916510 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x"] Oct 30 18:09:34.920794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.920757 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:34.937585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.936808 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:09:34.937585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.936865 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:09:34.937585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.936865 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:09:34.937585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.937004 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:09:34.938315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:34.938293 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x"] Oct 30 18:09:35.034745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.034716 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfgqn\" (UniqueName: \"kubernetes.io/projected/5ca83070-3d3d-49af-84ef-1a5c4108308a-kube-api-access-xfgqn\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.034958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.034909 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.035036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.035012 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.136186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.136155 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfgqn\" (UniqueName: \"kubernetes.io/projected/5ca83070-3d3d-49af-84ef-1a5c4108308a-kube-api-access-xfgqn\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.136387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.136195 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.136387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.136234 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.136979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.136955 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.164857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.140214 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.166107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.165807 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfgqn\" (UniqueName: \"kubernetes.io/projected/5ca83070-3d3d-49af-84ef-1a5c4108308a-kube-api-access-xfgqn\") pod \"maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.241071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.240983 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:35.392843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.376761 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:35.537476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.537383 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" event={"ID":"0bae5601-08f4-4f35-8b25-e8eda5a987b1","Type":"ContainerStarted","Data":"8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8"} Oct 30 18:09:35.540605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.540578 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" event={"ID":"cd594cec-adc6-48c7-aad6-45c444ce6c59","Type":"ContainerStarted","Data":"666a9b28bb7e50e22e0db789e3715ae8d9d13aa06380a0adbfb3d5a37edaca02"} Oct 30 18:09:35.637478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.637409 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:35.637628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.637596 4582 patch_prober.go:29] interesting pod/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:09:35.637674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.637645 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 18:09:35.738512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.738429 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x"] Oct 30 18:09:35.771855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:35.770765 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:09:36.317265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:36.317231 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:36.318017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:36.317992 4582 patch_prober.go:29] interesting pod/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:09:36.318180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:36.318155 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 18:09:36.321842 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:09:36.321509 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:09:36.596913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:36.595786 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" event={"ID":"5ca83070-3d3d-49af-84ef-1a5c4108308a","Type":"ContainerStarted","Data":"0fb10bb8c91f1c0b348fa3ed97f7f6f83ebbc093a963c156b16405ffda0f254f"} Oct 30 18:09:36.649858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:36.636877 4582 patch_prober.go:29] interesting pod/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:09:36.649858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:36.636931 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 18:09:37.116844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.116245 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:37.134106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.134065 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp"] Oct 30 18:09:37.136967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.136950 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.224854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.214905 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp"] Oct 30 18:09:37.240848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.240477 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.240848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.240534 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.240848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.240686 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmzj\" (UniqueName: \"kubernetes.io/projected/0ff046f1-5342-4f9b-9532-7dfea3764962-kube-api-access-hkmzj\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.319680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.319221 4582 patch_prober.go:29] interesting pod/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:09:37.319680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.319282 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 18:09:37.383927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.352348 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.383927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.352408 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.383927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.352561 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmzj\" (UniqueName: \"kubernetes.io/projected/0ff046f1-5342-4f9b-9532-7dfea3764962-kube-api-access-hkmzj\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.383927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.356879 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.383927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.365478 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.383927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.377065 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmzj\" (UniqueName: \"kubernetes.io/projected/0ff046f1-5342-4f9b-9532-7dfea3764962-kube-api-access-hkmzj\") pod \"maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.518159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.518122 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:37.636915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.636813 4582 patch_prober.go:29] interesting pod/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:09:37.636915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:37.636879 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 18:09:38.009807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:38.009687 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp"] Oct 30 18:09:38.025970 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:09:38.025432 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff046f1_5342_4f9b_9532_7dfea3764962.slice/crio-5d3e4cb7f07c6d9afa06b127416f950f4192d54b249d0181bd9357382b1bc8bb WatchSource:0}: Error finding container 5d3e4cb7f07c6d9afa06b127416f950f4192d54b249d0181bd9357382b1bc8bb: Status 404 returned error can't find the container with id 5d3e4cb7f07c6d9afa06b127416f950f4192d54b249d0181bd9357382b1bc8bb Oct 30 18:09:38.317900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:38.314884 4582 patch_prober.go:29] interesting pod/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:09:38.317900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:38.314951 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 18:09:38.637451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:38.637289 4582 patch_prober.go:29] interesting pod/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:09:38.637451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:38.637351 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 18:09:38.663245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:38.663154 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" event={"ID":"0ff046f1-5342-4f9b-9532-7dfea3764962","Type":"ContainerStarted","Data":"5d3e4cb7f07c6d9afa06b127416f950f4192d54b249d0181bd9357382b1bc8bb"} Oct 30 18:09:38.934893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:38.934791 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:39.035729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:39.035294 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:39.323911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:39.323314 4582 patch_prober.go:29] interesting pod/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:09:39.323911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:39.323370 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 18:09:39.637856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:39.637442 4582 patch_prober.go:29] interesting pod/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:09:39.637856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:39.637506 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 18:09:39.786850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:39.786392 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:40.138699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.138664 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm"] Oct 30 18:09:40.150783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.150752 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.178858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.163685 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm"] Oct 30 18:09:40.218857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.218804 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.219168 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.219144 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6l6w\" (UniqueName: \"kubernetes.io/projected/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-kube-api-access-g6l6w\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.219353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.219325 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.315856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.313923 4582 patch_prober.go:29] interesting pod/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 18:09:40.315856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.314020 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 18:09:40.324966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.324155 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.324966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.324241 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6l6w\" (UniqueName: \"kubernetes.io/projected/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-kube-api-access-g6l6w\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.324966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.324301 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.327041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.327012 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.335617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.335594 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.388622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.388551 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6l6w\" (UniqueName: \"kubernetes.io/projected/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-kube-api-access-g6l6w\") pod \"maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.535597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.535384 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:40.637371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.637050 4582 patch_prober.go:29] interesting pod/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:09:40.637371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.637099 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 30 18:09:40.774845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.764029 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:40.943782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:40.943522 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm"] Oct 30 18:09:41.316687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:41.315677 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:41.316687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:41.316584 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:41.317358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:41.317336 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:41.642661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:41.642630 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:41.643130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:41.643067 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:41.644123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:41.644100 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:09:41.754712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:41.754679 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" event={"ID":"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0","Type":"ContainerStarted","Data":"d3f854f74d4bf6b7ab4972aa7f0252c56daa303f9c9ee394a65ed59fa0dc1333"} Oct 30 18:09:42.584573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:42.584540 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:42.761276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:42.761246 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" event={"ID":"5ca83070-3d3d-49af-84ef-1a5c4108308a","Type":"ContainerStarted","Data":"9b6933f3f4b8de1cea82b6dbd37bfd8d9dd41e5e37f3db8b7c939c773588e87b"} Oct 30 18:09:42.799150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:42.799086 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" podStartSLOduration=2.495777087 podStartE2EDuration="8.799070449s" totalImagesPullingTime="6.303293362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:42.79554361 +0000 UTC m=+679.619937902" watchObservedRunningTime="2026-10-30 18:09:42.799070449 +0000 UTC m=+679.623464742" Oct 30 18:09:43.776679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:43.776644 4582 generic.go:356] "Generic (PLEG): container finished" podID="0ff046f1-5342-4f9b-9532-7dfea3764962" containerID="11243bba40d582086ccf0449ac6533a4e6df1797f1a788484ad92e2309085434" exitCode=0 Oct 30 18:09:43.777123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:43.776723 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" event={"ID":"0ff046f1-5342-4f9b-9532-7dfea3764962","Type":"ContainerDied","Data":"11243bba40d582086ccf0449ac6533a4e6df1797f1a788484ad92e2309085434"} Oct 30 18:09:43.778371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:43.778345 4582 generic.go:356] "Generic (PLEG): container finished" podID="8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" containerID="05655494b869893f770ae460feee1ec8eaf539a7454e3db507a7e1d80f87cba4" exitCode=0 Oct 30 18:09:43.778489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:43.778420 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" event={"ID":"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0","Type":"ContainerDied","Data":"05655494b869893f770ae460feee1ec8eaf539a7454e3db507a7e1d80f87cba4"} Oct 30 18:09:43.779953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:43.779930 4582 generic.go:356] "Generic (PLEG): container finished" podID="5ca83070-3d3d-49af-84ef-1a5c4108308a" containerID="9b6933f3f4b8de1cea82b6dbd37bfd8d9dd41e5e37f3db8b7c939c773588e87b" exitCode=0 Oct 30 18:09:43.780026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:43.780003 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" event={"ID":"5ca83070-3d3d-49af-84ef-1a5c4108308a","Type":"ContainerDied","Data":"9b6933f3f4b8de1cea82b6dbd37bfd8d9dd41e5e37f3db8b7c939c773588e87b"} Oct 30 18:09:44.157790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.157758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:44.297223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.295992 4582 patch_prober.go:29] interesting pod/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": read tcp 10.132.0.2:49228->10.132.0.44:15021: read: connection reset by peer" start-of-body= Oct 30 18:09:44.297223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.296053 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": read tcp 10.132.0.2:49228->10.132.0.44:15021: read: connection reset by peer" Oct 30 18:09:44.787194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.787161 4582 generic.go:356] "Generic (PLEG): container finished" podID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerID="a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235" exitCode=0 Oct 30 18:09:44.787690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.787223 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" event={"ID":"36b64be5-1cd8-46f8-989e-a79428dbae84","Type":"ContainerDied","Data":"a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235"} Oct 30 18:09:44.788485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.788037 4582 scope.go:122] "RemoveContainer" containerID="a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235" Oct 30 18:09:44.789633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.789597 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:44.789752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.789687 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:44.789752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:44.789731 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:45.102281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.100530 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:45.113621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.113424 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:45.116064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.115382 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:45.189236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189196 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-openshift-ca\") pod \"0ff046f1-5342-4f9b-9532-7dfea3764962\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " Oct 30 18:09:45.189422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189263 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-openshift-ca\") pod \"5ca83070-3d3d-49af-84ef-1a5c4108308a\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " Oct 30 18:09:45.189422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189293 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-xks-ca\") pod \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " Oct 30 18:09:45.189422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189339 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca83070-3d3d-49af-84ef-1a5c4108308a-kube-api-access-xfgqn\" (UniqueName: \"kubernetes.io/projected/5ca83070-3d3d-49af-84ef-1a5c4108308a-kube-api-access-xfgqn\") pod \"5ca83070-3d3d-49af-84ef-1a5c4108308a\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " Oct 30 18:09:45.189422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189383 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-xks-ca\") pod \"0ff046f1-5342-4f9b-9532-7dfea3764962\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " Oct 30 18:09:45.189422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189408 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-openshift-ca\") pod \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " Oct 30 18:09:45.189681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189441 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-kube-api-access-g6l6w\" (UniqueName: \"kubernetes.io/projected/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-kube-api-access-g6l6w\") pod \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\" (UID: \"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0\") " Oct 30 18:09:45.189681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189512 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff046f1-5342-4f9b-9532-7dfea3764962-kube-api-access-hkmzj\" (UniqueName: \"kubernetes.io/projected/0ff046f1-5342-4f9b-9532-7dfea3764962-kube-api-access-hkmzj\") pod \"0ff046f1-5342-4f9b-9532-7dfea3764962\" (UID: \"0ff046f1-5342-4f9b-9532-7dfea3764962\") " Oct 30 18:09:45.189681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.189584 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-xks-ca\") pod \"5ca83070-3d3d-49af-84ef-1a5c4108308a\" (UID: \"5ca83070-3d3d-49af-84ef-1a5c4108308a\") " Oct 30 18:09:45.190710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.190669 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-openshift-ca" pod "8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" (UID: "8dc7cdde-19ed-43b3-9691-6a731fd8a7b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:45.191902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.191502 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-openshift-ca" pod "5ca83070-3d3d-49af-84ef-1a5c4108308a" (UID: "5ca83070-3d3d-49af-84ef-1a5c4108308a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:45.191902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.191863 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-openshift-ca" pod "0ff046f1-5342-4f9b-9532-7dfea3764962" (UID: "0ff046f1-5342-4f9b-9532-7dfea3764962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:45.194265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.194239 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-xks-ca" pod "8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" (UID: "8dc7cdde-19ed-43b3-9691-6a731fd8a7b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:45.197840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.197792 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-kube-api-access-g6l6w" pod "8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" (UID: "8dc7cdde-19ed-43b3-9691-6a731fd8a7b0"). InnerVolumeSpecName "kube-api-access-g6l6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:45.197927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.197858 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff046f1-5342-4f9b-9532-7dfea3764962-kube-api-access-hkmzj" pod "0ff046f1-5342-4f9b-9532-7dfea3764962" (UID: "0ff046f1-5342-4f9b-9532-7dfea3764962"). InnerVolumeSpecName "kube-api-access-hkmzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:45.197927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.197867 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-xks-ca" pod "5ca83070-3d3d-49af-84ef-1a5c4108308a" (UID: "5ca83070-3d3d-49af-84ef-1a5c4108308a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:45.198429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.198402 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-xks-ca" pod "0ff046f1-5342-4f9b-9532-7dfea3764962" (UID: "0ff046f1-5342-4f9b-9532-7dfea3764962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:45.199614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.199588 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca83070-3d3d-49af-84ef-1a5c4108308a-kube-api-access-xfgqn" pod "5ca83070-3d3d-49af-84ef-1a5c4108308a" (UID: "5ca83070-3d3d-49af-84ef-1a5c4108308a"). InnerVolumeSpecName "kube-api-access-xfgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:45.291355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291312 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291346 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291356 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291365 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfgqn\" (UniqueName: \"kubernetes.io/projected/5ca83070-3d3d-49af-84ef-1a5c4108308a-kube-api-access-xfgqn\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291375 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff046f1-5342-4f9b-9532-7dfea3764962-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291383 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291391 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6l6w\" (UniqueName: \"kubernetes.io/projected/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0-kube-api-access-g6l6w\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291401 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkmzj\" (UniqueName: \"kubernetes.io/projected/0ff046f1-5342-4f9b-9532-7dfea3764962-kube-api-access-hkmzj\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.291611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.291409 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca83070-3d3d-49af-84ef-1a5c4108308a-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:45.729784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.729745 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:45.812441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.812407 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" event={"ID":"0ff046f1-5342-4f9b-9532-7dfea3764962","Type":"ContainerDied","Data":"5d3e4cb7f07c6d9afa06b127416f950f4192d54b249d0181bd9357382b1bc8bb"} Oct 30 18:09:45.812850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.812468 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d3e4cb7f07c6d9afa06b127416f950f4192d54b249d0181bd9357382b1bc8bb" Oct 30 18:09:45.812850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.812533 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp" Oct 30 18:09:45.815194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.814782 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" Oct 30 18:09:45.815194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.814794 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm" event={"ID":"8dc7cdde-19ed-43b3-9691-6a731fd8a7b0","Type":"ContainerDied","Data":"d3f854f74d4bf6b7ab4972aa7f0252c56daa303f9c9ee394a65ed59fa0dc1333"} Oct 30 18:09:45.815194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.814862 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3f854f74d4bf6b7ab4972aa7f0252c56daa303f9c9ee394a65ed59fa0dc1333" Oct 30 18:09:45.816567 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.816551 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" Oct 30 18:09:45.816849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.816573 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x" event={"ID":"5ca83070-3d3d-49af-84ef-1a5c4108308a","Type":"ContainerDied","Data":"0fb10bb8c91f1c0b348fa3ed97f7f6f83ebbc093a963c156b16405ffda0f254f"} Oct 30 18:09:45.816962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.816869 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fb10bb8c91f1c0b348fa3ed97f7f6f83ebbc093a963c156b16405ffda0f254f" Oct 30 18:09:45.819244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:45.819208 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" event={"ID":"36b64be5-1cd8-46f8-989e-a79428dbae84","Type":"ContainerStarted","Data":"49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f"} Oct 30 18:09:46.518066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:46.518034 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:46.518394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:46.518370 4582 patch_prober.go:29] interesting pod/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:46.518489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:46.518426 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 18:09:47.259887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:47.259757 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:47.518460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:47.518371 4582 patch_prober.go:29] interesting pod/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:47.518720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:47.518691 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 18:09:47.858981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:47.858888 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:47.936087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:47.936052 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:48.518804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:48.518769 4582 patch_prober.go:29] interesting pod/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:48.519199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:48.518849 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 18:09:48.860211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:48.860120 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:49.518026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:49.517988 4582 patch_prober.go:29] interesting pod/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:49.518208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:49.518041 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 18:09:50.226474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.226445 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:50.518295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.518215 4582 patch_prober.go:29] interesting pod/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:50.518460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.518285 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 18:09:50.684046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.684015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:50.685868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.685839 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk"] Oct 30 18:09:50.686644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686617 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0ff046f1-5342-4f9b-9532-7dfea3764962" containerName="revoke-keys" Oct 30 18:09:50.686644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686643 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff046f1-5342-4f9b-9532-7dfea3764962" containerName="revoke-keys" Oct 30 18:09:50.686814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686671 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5ca83070-3d3d-49af-84ef-1a5c4108308a" containerName="revoke-keys" Oct 30 18:09:50.686814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686680 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca83070-3d3d-49af-84ef-1a5c4108308a" containerName="revoke-keys" Oct 30 18:09:50.686814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686716 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" containerName="revoke-keys" Oct 30 18:09:50.686814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686725 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" containerName="revoke-keys" Oct 30 18:09:50.686963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686862 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff046f1-5342-4f9b-9532-7dfea3764962" containerName="revoke-keys" Oct 30 18:09:50.686963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686877 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca83070-3d3d-49af-84ef-1a5c4108308a" containerName="revoke-keys" Oct 30 18:09:50.686963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.686891 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" containerName="revoke-keys" Oct 30 18:09:50.693762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.693737 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.705064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.705036 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk"] Oct 30 18:09:50.710782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.710762 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:09:50.710947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.710856 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:09:50.712042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.712009 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:09:50.718151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.718113 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:09:50.780405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.780320 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:50.807257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.807204 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtvf\" (UniqueName: \"kubernetes.io/projected/d95ceba4-6f63-40eb-9e9a-24fd98de9102-kube-api-access-whtvf\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.807462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.807373 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.807462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.807407 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.908761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.908725 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.908947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.908772 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.909003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.908981 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtvf\" (UniqueName: \"kubernetes.io/projected/d95ceba4-6f63-40eb-9e9a-24fd98de9102-kube-api-access-whtvf\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.909416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.909393 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.911331 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.911298 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:50.918476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.918451 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:50.924908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:50.924888 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtvf\" (UniqueName: \"kubernetes.io/projected/d95ceba4-6f63-40eb-9e9a-24fd98de9102-kube-api-access-whtvf\") pod \"maas-api-revoke-keys-e2e-ait-54599e13-9rhjk\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:51.014879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.014847 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:51.018785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.018742 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:51.109633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.109602 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp"] Oct 30 18:09:51.121147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.121115 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:51.121479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.121449 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a521cf49-vmzsp"] Oct 30 18:09:51.197175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.197136 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq"] Oct 30 18:09:51.229583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.229553 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.230112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.230089 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq"] Oct 30 18:09:51.299963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.299888 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff046f1-5342-4f9b-9532-7dfea3764962" path="/var/lib/kubelet/pods/0ff046f1-5342-4f9b-9532-7dfea3764962/volumes" Oct 30 18:09:51.320555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.320503 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.320734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.320631 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.320801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.320749 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nbx\" (UniqueName: \"kubernetes.io/projected/845566b0-bca8-47f6-98b0-6b679053e960-kube-api-access-r8nbx\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.330926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.330895 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk"] Oct 30 18:09:51.334577 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:09:51.334551 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95ceba4_6f63_40eb_9e9a_24fd98de9102.slice/crio-70403792641e2b5488c83def4e9aaaf2fa43637ed3be7f97224ce66124683bba WatchSource:0}: Error finding container 70403792641e2b5488c83def4e9aaaf2fa43637ed3be7f97224ce66124683bba: Status 404 returned error can't find the container with id 70403792641e2b5488c83def4e9aaaf2fa43637ed3be7f97224ce66124683bba Oct 30 18:09:51.421390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.421345 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.421579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.421410 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.421579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.421465 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nbx\" (UniqueName: \"kubernetes.io/projected/845566b0-bca8-47f6-98b0-6b679053e960-kube-api-access-r8nbx\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.422516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.422300 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.423741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.423718 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.442440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.442404 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nbx\" (UniqueName: \"kubernetes.io/projected/845566b0-bca8-47f6-98b0-6b679053e960-kube-api-access-r8nbx\") pod \"maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.522632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.522595 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:51.523181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.523151 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:51.523961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.523943 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:51.545085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.545060 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:51.709666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.709623 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x"] Oct 30 18:09:51.710785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.710522 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" containerID="cri-o://666a9b28bb7e50e22e0db789e3715ae8d9d13aa06380a0adbfb3d5a37edaca02" gracePeriod=30 Oct 30 18:09:51.733416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.733337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:51.855521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.855436 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq"] Oct 30 18:09:51.859953 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:09:51.859921 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845566b0_bca8_47f6_98b0_6b679053e960.slice/crio-c58e018b1d96bacaa4b339e5a44cac33ef0386dc126ad28bfc6d0b44358f8bea WatchSource:0}: Error finding container c58e018b1d96bacaa4b339e5a44cac33ef0386dc126ad28bfc6d0b44358f8bea: Status 404 returned error can't find the container with id c58e018b1d96bacaa4b339e5a44cac33ef0386dc126ad28bfc6d0b44358f8bea Oct 30 18:09:51.876368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.876329 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" event={"ID":"d95ceba4-6f63-40eb-9e9a-24fd98de9102","Type":"ContainerStarted","Data":"5c6613b3ba3a8238aeb388448222da3b01dd96b8884a8f206956ea8ee577eb71"} Oct 30 18:09:51.876501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.876414 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" event={"ID":"d95ceba4-6f63-40eb-9e9a-24fd98de9102","Type":"ContainerStarted","Data":"70403792641e2b5488c83def4e9aaaf2fa43637ed3be7f97224ce66124683bba"} Oct 30 18:09:51.878166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.878097 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" event={"ID":"845566b0-bca8-47f6-98b0-6b679053e960","Type":"ContainerStarted","Data":"c58e018b1d96bacaa4b339e5a44cac33ef0386dc126ad28bfc6d0b44358f8bea"} Oct 30 18:09:51.909831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:51.909777 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" podStartSLOduration=1.90976338 podStartE2EDuration="1.90976338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:51.908016234 +0000 UTC m=+688.732410526" watchObservedRunningTime="2026-10-30 18:09:51.90976338 +0000 UTC m=+688.734157672" Oct 30 18:09:52.335734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:52.335694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:52.545233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:52.545122 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:52.884698 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:52.884667 4582 generic.go:356] "Generic (PLEG): container finished" podID="d95ceba4-6f63-40eb-9e9a-24fd98de9102" containerID="5c6613b3ba3a8238aeb388448222da3b01dd96b8884a8f206956ea8ee577eb71" exitCode=0 Oct 30 18:09:52.884867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:52.884760 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" event={"ID":"d95ceba4-6f63-40eb-9e9a-24fd98de9102","Type":"ContainerDied","Data":"5c6613b3ba3a8238aeb388448222da3b01dd96b8884a8f206956ea8ee577eb71"} Oct 30 18:09:52.886425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:52.886396 4582 generic.go:356] "Generic (PLEG): container finished" podID="845566b0-bca8-47f6-98b0-6b679053e960" containerID="53effe18ddb26a172588ce750f5e6b2ea1b75be6ccccfc387b9799829f10addc" exitCode=0 Oct 30 18:09:52.886509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:52.886425 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" event={"ID":"845566b0-bca8-47f6-98b0-6b679053e960","Type":"ContainerDied","Data":"53effe18ddb26a172588ce750f5e6b2ea1b75be6ccccfc387b9799829f10addc"} Oct 30 18:09:52.926295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:52.926265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.040916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.040473 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.089933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.089899 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm"] Oct 30 18:09:53.090674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.090653 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-594f46d3-qnrlm"] Oct 30 18:09:53.130101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.130064 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.232015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.231910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.239859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.238799 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw"] Oct 30 18:09:53.301662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.301635 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc7cdde-19ed-43b3-9691-6a731fd8a7b0" path="/var/lib/kubelet/pods/8dc7cdde-19ed-43b3-9691-6a731fd8a7b0/volumes" Oct 30 18:09:53.327296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.327273 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.893458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.893381 4582 generic.go:356] "Generic (PLEG): container finished" podID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerID="55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9" exitCode=0 Oct 30 18:09:53.893980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.893456 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" event={"ID":"dcb2e704-471b-4172-98e8-10ec56b4f5a6","Type":"ContainerDied","Data":"55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9"} Oct 30 18:09:53.893980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.893623 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" containerID="cri-o://49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f" gracePeriod=30 Oct 30 18:09:53.894602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.894321 4582 scope.go:122] "RemoveContainer" containerID="55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9" Oct 30 18:09:53.895654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.895619 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:53.895753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.895709 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:53.895811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:53.895757 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:09:54.073850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.073811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:54.077199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.077178 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:54.083339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.083314 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:54.260618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.260533 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-openshift-ca\") pod \"845566b0-bca8-47f6-98b0-6b679053e960\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " Oct 30 18:09:54.260776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.260618 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-openshift-ca\") pod \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " Oct 30 18:09:54.260776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.260721 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95ceba4-6f63-40eb-9e9a-24fd98de9102-kube-api-access-whtvf\" (UniqueName: \"kubernetes.io/projected/d95ceba4-6f63-40eb-9e9a-24fd98de9102-kube-api-access-whtvf\") pod \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " Oct 30 18:09:54.260931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.260785 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-xks-ca\") pod \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\" (UID: \"d95ceba4-6f63-40eb-9e9a-24fd98de9102\") " Oct 30 18:09:54.260931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.260857 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845566b0-bca8-47f6-98b0-6b679053e960-kube-api-access-r8nbx\" (UniqueName: \"kubernetes.io/projected/845566b0-bca8-47f6-98b0-6b679053e960-kube-api-access-r8nbx\") pod \"845566b0-bca8-47f6-98b0-6b679053e960\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " Oct 30 18:09:54.260931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.260890 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-xks-ca\") pod \"845566b0-bca8-47f6-98b0-6b679053e960\" (UID: \"845566b0-bca8-47f6-98b0-6b679053e960\") " Oct 30 18:09:54.261146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.261113 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-openshift-ca" pod "d95ceba4-6f63-40eb-9e9a-24fd98de9102" (UID: "d95ceba4-6f63-40eb-9e9a-24fd98de9102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:54.261298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.261279 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-openshift-ca" pod "845566b0-bca8-47f6-98b0-6b679053e960" (UID: "845566b0-bca8-47f6-98b0-6b679053e960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:54.261632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.261610 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.261760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.261745 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.263250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.263225 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-xks-ca" pod "845566b0-bca8-47f6-98b0-6b679053e960" (UID: "845566b0-bca8-47f6-98b0-6b679053e960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:54.263453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.263433 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845566b0-bca8-47f6-98b0-6b679053e960-kube-api-access-r8nbx" pod "845566b0-bca8-47f6-98b0-6b679053e960" (UID: "845566b0-bca8-47f6-98b0-6b679053e960"). InnerVolumeSpecName "kube-api-access-r8nbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:54.263556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.263538 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-xks-ca" pod "d95ceba4-6f63-40eb-9e9a-24fd98de9102" (UID: "d95ceba4-6f63-40eb-9e9a-24fd98de9102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:54.263610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.263592 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95ceba4-6f63-40eb-9e9a-24fd98de9102-kube-api-access-whtvf" pod "d95ceba4-6f63-40eb-9e9a-24fd98de9102" (UID: "d95ceba4-6f63-40eb-9e9a-24fd98de9102"). InnerVolumeSpecName "kube-api-access-whtvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:54.363877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.363305 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtvf\" (UniqueName: \"kubernetes.io/projected/d95ceba4-6f63-40eb-9e9a-24fd98de9102-kube-api-access-whtvf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.363877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.363346 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ceba4-6f63-40eb-9e9a-24fd98de9102-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.363877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.363361 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8nbx\" (UniqueName: \"kubernetes.io/projected/845566b0-bca8-47f6-98b0-6b679053e960-kube-api-access-r8nbx\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.363877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.363374 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845566b0-bca8-47f6-98b0-6b679053e960-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.571935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.569676 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:54.793988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.793954 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:54.900420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.900389 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" Oct 30 18:09:54.900420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.900406 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk" event={"ID":"d95ceba4-6f63-40eb-9e9a-24fd98de9102","Type":"ContainerDied","Data":"70403792641e2b5488c83def4e9aaaf2fa43637ed3be7f97224ce66124683bba"} Oct 30 18:09:54.900957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.900453 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70403792641e2b5488c83def4e9aaaf2fa43637ed3be7f97224ce66124683bba" Oct 30 18:09:54.902454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.902421 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" event={"ID":"dcb2e704-471b-4172-98e8-10ec56b4f5a6","Type":"ContainerStarted","Data":"017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598"} Oct 30 18:09:54.903794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.903775 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" Oct 30 18:09:54.903933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.903788 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq" event={"ID":"845566b0-bca8-47f6-98b0-6b679053e960","Type":"ContainerDied","Data":"c58e018b1d96bacaa4b339e5a44cac33ef0386dc126ad28bfc6d0b44358f8bea"} Oct 30 18:09:54.903933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:54.903858 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c58e018b1d96bacaa4b339e5a44cac33ef0386dc126ad28bfc6d0b44358f8bea" Oct 30 18:09:55.171327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:55.171241 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:55.599614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:55.599526 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:55.611445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:55.611415 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:09:55.611804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:55.611786 4582 patch_prober.go:29] interesting pod/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:09:55.611917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:55.611841 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:09:55.698384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:55.698353 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:56.318604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.318535 4582 patch_prober.go:29] interesting pod/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:09:56.318992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.318623 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:09:56.575307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.575218 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:56.610892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.610859 4582 patch_prober.go:29] interesting pod/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:09:56.611040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.610913 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:09:56.675995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.675955 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:56.918634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.918601 4582 generic.go:356] "Generic (PLEG): container finished" podID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerID="666a9b28bb7e50e22e0db789e3715ae8d9d13aa06380a0adbfb3d5a37edaca02" exitCode=0 Oct 30 18:09:56.918857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.918624 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" event={"ID":"cd594cec-adc6-48c7-aad6-45c444ce6c59","Type":"ContainerDied","Data":"666a9b28bb7e50e22e0db789e3715ae8d9d13aa06380a0adbfb3d5a37edaca02"} Oct 30 18:09:56.918857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:56.918702 4582 scope.go:122] "RemoveContainer" containerID="469d3fae1330268bc49b420546b608c8c6aca797ff8b5fea28a04f0047093122" Oct 30 18:09:57.002508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.002485 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:57.028873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.028848 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:57.195951 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.195915 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-kube-api-access-mfc9l\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-kube-api-access-mfc9l\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.195966 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-data\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196027 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-envoy\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196072 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-socket\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196151 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-podinfo\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196207 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-token\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-token\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196241 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd594cec-adc6-48c7-aad6-45c444ce6c59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd594cec-adc6-48c7-aad6-45c444ce6c59-istiod-ca-cert\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196341 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-data" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:57.196522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196461 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-socket" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:57.196522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196485 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-credential-socket\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196596 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-certs\") pod \"cd594cec-adc6-48c7-aad6-45c444ce6c59\" (UID: \"cd594cec-adc6-48c7-aad6-45c444ce6c59\") " Oct 30 18:09:57.196742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196718 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-credential-socket" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:57.196845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196766 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd594cec-adc6-48c7-aad6-45c444ce6c59-istiod-ca-cert" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:57.196916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.196893 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-certs" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:57.197152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.197127 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd594cec-adc6-48c7-aad6-45c444ce6c59-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.197237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.197158 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.197237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.197177 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.197237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.197192 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.197237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.197205 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.198522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.198498 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-token" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:57.198743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.198721 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-kube-api-access-mfc9l" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "kube-api-access-mfc9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:57.198872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.198769 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-envoy" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:57.198927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.198871 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-podinfo" pod "cd594cec-adc6-48c7-aad6-45c444ce6c59" (UID: "cd594cec-adc6-48c7-aad6-45c444ce6c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:09:57.298058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.298028 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfc9l\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-kube-api-access-mfc9l\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.298058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.298054 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.298058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.298065 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.298286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.298073 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd594cec-adc6-48c7-aad6-45c444ce6c59-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:57.611247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.611161 4582 patch_prober.go:29] interesting pod/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:09:57.611247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.611220 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:09:57.925536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.925507 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" Oct 30 18:09:57.925706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.925536 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x" event={"ID":"cd594cec-adc6-48c7-aad6-45c444ce6c59","Type":"ContainerDied","Data":"827c3b9e7df9efa53239c3e9f94b766d7c784122db3f39e89c702adecbb364be"} Oct 30 18:09:57.925706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.925610 4582 scope.go:122] "RemoveContainer" containerID="666a9b28bb7e50e22e0db789e3715ae8d9d13aa06380a0adbfb3d5a37edaca02" Oct 30 18:09:57.986966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.986933 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x"] Oct 30 18:09:57.996676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:57.996646 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a521cf49-openshift-default-d59b5bdbc-nlz9x"] Oct 30 18:09:58.336055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:58.335957 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:58.469038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:58.469002 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:58.569406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:58.569367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:58.611551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:58.611464 4582 patch_prober.go:29] interesting pod/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:09:58.611551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:58.611514 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:09:58.675425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:58.675397 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.193783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.193760 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:59.266712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.265217 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.299290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.299250 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" path="/var/lib/kubelet/pods/cd594cec-adc6-48c7-aad6-45c444ce6c59/volumes" Oct 30 18:09:59.319377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319354 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b64be5-1cd8-46f8-989e-a79428dbae84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b64be5-1cd8-46f8-989e-a79428dbae84-istiod-ca-cert\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319420 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-envoy\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319443 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-socket\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319476 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-data\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319540 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-kube-api-access-ldqlh\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-kube-api-access-ldqlh\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319562 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-token\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-token\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319626 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-credential-socket\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319683 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-certs\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319721 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-podinfo\") pod \"36b64be5-1cd8-46f8-989e-a79428dbae84\" (UID: \"36b64be5-1cd8-46f8-989e-a79428dbae84\") " Oct 30 18:09:59.319924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319722 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b64be5-1cd8-46f8-989e-a79428dbae84-istiod-ca-cert" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:59.319924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319781 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-socket" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:59.319924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.319895 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-data" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:59.320132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.320002 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-credential-socket" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:59.320132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.320094 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-certs" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:59.320243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.320192 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.320243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.320205 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.320243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.320214 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b64be5-1cd8-46f8-989e-a79428dbae84-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.320243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.320223 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.320243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.320232 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.321748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.321726 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-envoy" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:59.321899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.321883 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-kube-api-access-ldqlh" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "kube-api-access-ldqlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:59.322014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.321997 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-token" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:59.322072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.322000 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-podinfo" pod "36b64be5-1cd8-46f8-989e-a79428dbae84" (UID: "36b64be5-1cd8-46f8-989e-a79428dbae84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:09:59.367800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.367765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.403695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.403662 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x"] Oct 30 18:09:59.406090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.406063 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57bf5177-drq2x"] Oct 30 18:09:59.421247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.421176 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqlh\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-kube-api-access-ldqlh\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.421247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.421204 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.421247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.421216 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.421247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.421226 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b64be5-1cd8-46f8-989e-a79428dbae84-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:09:59.463469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.463313 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.572639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.572607 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.611809 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.611776 4582 patch_prober.go:29] interesting pod/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:09:59.612247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.611870 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:09:59.944551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.944517 4582 generic.go:356] "Generic (PLEG): container finished" podID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerID="49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f" exitCode=0 Oct 30 18:09:59.944713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.944561 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" Oct 30 18:09:59.944713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.944600 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" event={"ID":"36b64be5-1cd8-46f8-989e-a79428dbae84","Type":"ContainerDied","Data":"49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f"} Oct 30 18:09:59.944713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.944661 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw" event={"ID":"36b64be5-1cd8-46f8-989e-a79428dbae84","Type":"ContainerDied","Data":"5ea3ea8a9f6655ac9f1540de27919214893404cb9b69d45504b8f01ff670020a"} Oct 30 18:09:59.944713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.944691 4582 scope.go:122] "RemoveContainer" containerID="49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f" Oct 30 18:09:59.955859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.955836 4582 scope.go:122] "RemoveContainer" containerID="a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235" Oct 30 18:09:59.964192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.964172 4582 scope.go:122] "RemoveContainer" containerID="49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f" Oct 30 18:09:59.964475 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:09:59.964446 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f\": container with ID starting with 49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f not found: ID does not exist" containerID="49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f" Oct 30 18:09:59.964556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.964478 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f"} err="failed to get container status \"49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f\": rpc error: code = NotFound desc = could not find container \"49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f\": container with ID starting with 49daa6b675def1b257bf5449580c22e090a2d494d43f928a481bf2e30a7bef2f not found: ID does not exist" Oct 30 18:09:59.964556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.964505 4582 scope.go:122] "RemoveContainer" containerID="a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235" Oct 30 18:09:59.964776 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:09:59.964757 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235\": container with ID starting with a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235 not found: ID does not exist" containerID="a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235" Oct 30 18:09:59.964812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.964784 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235"} err="failed to get container status \"a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235\": rpc error: code = NotFound desc = could not find container \"a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235\": container with ID starting with a0c7c2e5732dfc8d51c2c958e166306d19424b3553f0f3308b67824870910235 not found: ID does not exist" Oct 30 18:09:59.989385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.987538 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw"] Oct 30 18:09:59.996652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:09:59.996623 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-594f46d3-openshift-default-7b6784c8cf-tcwsw"] Oct 30 18:10:00.291497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.291422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:00.387962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.387910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:00.491259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.491227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:00.615488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.615413 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:10:00.615952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.615795 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:10:00.616513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.616494 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:10:00.913677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.913643 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8"] Oct 30 18:10:00.914388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914366 4582 cpu_manager.go:414] "containerMap: removing container" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" Oct 30 18:10:00.914388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914387 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914412 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914420 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914446 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914454 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914467 4582 cpu_manager.go:414] "containerMap: removing container" podUID="845566b0-bca8-47f6-98b0-6b679053e960" containerName="revoke-keys" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914473 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845566b0-bca8-47f6-98b0-6b679053e960" containerName="revoke-keys" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914484 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d95ceba4-6f63-40eb-9e9a-24fd98de9102" containerName="revoke-keys" Oct 30 18:10:00.914555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914492 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95ceba4-6f63-40eb-9e9a-24fd98de9102" containerName="revoke-keys" Oct 30 18:10:00.914937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914586 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" Oct 30 18:10:00.914937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914596 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" Oct 30 18:10:00.914937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914604 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="845566b0-bca8-47f6-98b0-6b679053e960" containerName="revoke-keys" Oct 30 18:10:00.914937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.914612 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95ceba4-6f63-40eb-9e9a-24fd98de9102" containerName="revoke-keys" Oct 30 18:10:00.918382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.918359 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.930975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.930954 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-de7e3a-openshift-default-dockercfg-p5ml9\"" Oct 30 18:10:00.942446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.942418 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03abb5af-c9ed-4ba5-b54c-04752276b68f-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.942704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.942669 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.942779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.942723 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6f2s\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-kube-api-access-q6f2s\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.942989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.942966 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.943077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.943013 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.943077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.943071 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.943185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.943132 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.943239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.943188 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.943239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.943220 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:00.944917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.944847 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8"] Oct 30 18:10:00.966879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:00.966846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:01.044702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.044675 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.044864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.044707 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.044864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.044749 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.044996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.044899 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.044996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.044966 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.044996 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045050 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03abb5af-c9ed-4ba5-b54c-04752276b68f-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045114 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045163 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045189 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6f2s\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-kube-api-access-q6f2s\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045257 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045402 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045487 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.045780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.045760 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03abb5af-c9ed-4ba5-b54c-04752276b68f-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.047207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.047186 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.047411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.047393 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.064952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.064924 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.065076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.065058 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6f2s\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-kube-api-access-q6f2s\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.067468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.067435 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:01.180785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.180690 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:01.232917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.232887 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:01.303462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.303438 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" path="/var/lib/kubelet/pods/36b64be5-1cd8-46f8-989e-a79428dbae84/volumes" Oct 30 18:10:01.303889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.303873 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca83070-3d3d-49af-84ef-1a5c4108308a" path="/var/lib/kubelet/pods/5ca83070-3d3d-49af-84ef-1a5c4108308a/volumes" Oct 30 18:10:01.549082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.549004 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8"] Oct 30 18:10:01.553064 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:10:01.553037 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03abb5af_c9ed_4ba5_b54c_04752276b68f.slice/crio-60b1f2ce61bbe36540a5f26bf32c27b037dccdba3b911b3822554b8a40537331 WatchSource:0}: Error finding container 60b1f2ce61bbe36540a5f26bf32c27b037dccdba3b911b3822554b8a40537331: Status 404 returned error can't find the container with id 60b1f2ce61bbe36540a5f26bf32c27b037dccdba3b911b3822554b8a40537331 Oct 30 18:10:01.555414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.555381 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:01.555516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.555451 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:01.555516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.555483 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:01.577937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.577915 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:01.684917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.679695 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:01.959501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.959467 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" event={"ID":"03abb5af-c9ed-4ba5-b54c-04752276b68f","Type":"ContainerStarted","Data":"a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1"} Oct 30 18:10:01.959701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.959547 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" event={"ID":"03abb5af-c9ed-4ba5-b54c-04752276b68f","Type":"ContainerStarted","Data":"60b1f2ce61bbe36540a5f26bf32c27b037dccdba3b911b3822554b8a40537331"} Oct 30 18:10:01.997852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:01.997792 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podStartSLOduration=1.997778168 podStartE2EDuration="1.997778168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:01.994398954 +0000 UTC m=+698.818793257" watchObservedRunningTime="2026-10-30 18:10:01.997778168 +0000 UTC m=+698.822172458" Oct 30 18:10:02.233895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:02.233795 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:02.240436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:02.240414 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:02.337863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:02.337794 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:02.438266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:02.438224 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:02.964694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:02.964664 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:02.965816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:02.965795 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:03.028728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:03.028693 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:03.121520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:03.121487 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:03.224556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:03.224332 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:04.413789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:04.413756 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:04.516231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:04.516190 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:04.760732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:04.760661 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:05.267439 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.267400 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:05.437086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.437058 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:05.506348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.506317 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq"] Oct 30 18:10:05.515606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.515575 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ba56e909-5m2lq"] Oct 30 18:10:05.564322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.564240 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:05.659982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.659946 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9"] Oct 30 18:10:05.660508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.660474 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" containerID="cri-o://017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598" gracePeriod=30 Oct 30 18:10:05.668435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.668407 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:05.768307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:05.768269 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:06.224258 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:10:06.224224 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:10:06.486700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.486618 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:06.683580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.683550 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:06.783680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.783566 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:06.946109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.946075 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9"] Oct 30 18:10:06.946775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.946758 4582 cpu_manager.go:414] "containerMap: removing container" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" Oct 30 18:10:06.946869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.946777 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" Oct 30 18:10:06.946975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.946962 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b64be5-1cd8-46f8-989e-a79428dbae84" containerName="istio-proxy" Oct 30 18:10:06.947027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.946979 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd594cec-adc6-48c7-aad6-45c444ce6c59" containerName="istio-proxy" Oct 30 18:10:06.949752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.949730 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:06.974199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.974172 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9"] Oct 30 18:10:06.987067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.987044 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-380c08-openshift-default-dockercfg-74pzf\"" Oct 30 18:10:06.999276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:06.999238 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:07.010727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.010696 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.010845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.010770 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.010914 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.010816 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.010914 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.010886 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llt4h\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-kube-api-access-llt4h\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.011029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.010921 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.011029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.010968 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.011113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.011039 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.011113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.011060 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.011113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.011089 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.103509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.103419 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:07.111757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.111725 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.111913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.111776 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.111913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.111838 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llt4h\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-kube-api-access-llt4h\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.111913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.111867 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.111913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.111900 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.111981 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112005 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112034 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112098 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112200 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112243 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112719 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112717 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.112899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.112777 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.114535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.114517 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.114604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.114586 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.129578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.129553 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.130520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.130491 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llt4h\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-kube-api-access-llt4h\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.203370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.203339 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:07.265188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.265161 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:07.299948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.299917 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845566b0-bca8-47f6-98b0-6b679053e960" path="/var/lib/kubelet/pods/845566b0-bca8-47f6-98b0-6b679053e960/volumes" Oct 30 18:10:07.582416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.582378 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9"] Oct 30 18:10:07.588457 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:10:07.588415 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b1e0ec_6439_40fe_b77b_474d327d4cc9.slice/crio-93506dd48ece1ca4e853ecd9cecc030e2726e9c7637518bf29631a2c857d490d WatchSource:0}: Error finding container 93506dd48ece1ca4e853ecd9cecc030e2726e9c7637518bf29631a2c857d490d: Status 404 returned error can't find the container with id 93506dd48ece1ca4e853ecd9cecc030e2726e9c7637518bf29631a2c857d490d Oct 30 18:10:07.590785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.590746 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:07.590933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.590858 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:07.591000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.590944 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:07.803797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.803765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:07.980564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.980533 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:07.997033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.996992 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" event={"ID":"98b1e0ec-6439-40fe-b77b-474d327d4cc9","Type":"ContainerStarted","Data":"434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f"} Oct 30 18:10:07.997204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:07.997053 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" event={"ID":"98b1e0ec-6439-40fe-b77b-474d327d4cc9","Type":"ContainerStarted","Data":"93506dd48ece1ca4e853ecd9cecc030e2726e9c7637518bf29631a2c857d490d"} Oct 30 18:10:08.031537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:08.031506 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk"] Oct 30 18:10:08.040678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:08.040629 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podStartSLOduration=2.040615648 podStartE2EDuration="2.040615648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:08.038905872 +0000 UTC m=+704.863300166" watchObservedRunningTime="2026-10-30 18:10:08.040615648 +0000 UTC m=+704.865009940" Oct 30 18:10:08.043706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:08.043679 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54599e13-9rhjk"] Oct 30 18:10:08.098860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:08.098810 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:08.266016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:08.265937 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:08.271312 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:08.271280 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:09.002767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:09.002736 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:09.003769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:09.003751 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:09.299762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:09.299681 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95ceba4-6f63-40eb-9e9a-24fd98de9102" path="/var/lib/kubelet/pods/d95ceba4-6f63-40eb-9e9a-24fd98de9102/volumes" Oct 30 18:10:09.712850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:09.712800 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5"] Oct 30 18:10:09.713105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:09.713075 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" containerID="cri-o://8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8" gracePeriod=30 Oct 30 18:10:09.728765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:09.728741 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:09.914730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:09.914695 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:10.014065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.013988 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:10.114939 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.114906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:10.482369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.482337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:10.608476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.608443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:10.707247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.707219 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:10.921040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.921016 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:10:10.999255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.999219 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr"] Oct 30 18:10:10.999947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.999915 4582 cpu_manager.go:414] "containerMap: removing container" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" Oct 30 18:10:10.999947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.999936 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" Oct 30 18:10:11.000137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.999953 4582 cpu_manager.go:414] "containerMap: removing container" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" Oct 30 18:10:11.000137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:10.999961 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" Oct 30 18:10:11.000137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.000111 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" Oct 30 18:10:11.000137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.000126 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerName="istio-proxy" Oct 30 18:10:11.002487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.002469 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.015094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.015069 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fe1374-openshift-default-dockercfg-dg2jw\"" Oct 30 18:10:11.018289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.018265 4582 generic.go:356] "Generic (PLEG): container finished" podID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" containerID="017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598" exitCode=0 Oct 30 18:10:11.018412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.018320 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" Oct 30 18:10:11.018412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.018352 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" event={"ID":"dcb2e704-471b-4172-98e8-10ec56b4f5a6","Type":"ContainerDied","Data":"017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598"} Oct 30 18:10:11.018412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.018405 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9" event={"ID":"dcb2e704-471b-4172-98e8-10ec56b4f5a6","Type":"ContainerDied","Data":"7fd97bd0732b6fb1eacb3c6d7bfd6e32017caa4816dd94a74a8cf7dfe9f8e797"} Oct 30 18:10:11.018584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.018430 4582 scope.go:122] "RemoveContainer" containerID="017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598" Oct 30 18:10:11.026304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.026283 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr"] Oct 30 18:10:11.030475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.030456 4582 scope.go:122] "RemoveContainer" containerID="55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9" Oct 30 18:10:11.041164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.041145 4582 scope.go:122] "RemoveContainer" containerID="017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598" Oct 30 18:10:11.041461 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:10:11.041431 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598\": container with ID starting with 017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598 not found: ID does not exist" containerID="017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598" Oct 30 18:10:11.041556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.041470 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598"} err="failed to get container status \"017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598\": rpc error: code = NotFound desc = could not find container \"017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598\": container with ID starting with 017f1794c640b4385b2b830e595e0ffa1e40c87fd31b139556539cbc50ba4598 not found: ID does not exist" Oct 30 18:10:11.041556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.041493 4582 scope.go:122] "RemoveContainer" containerID="55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9" Oct 30 18:10:11.041800 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:10:11.041777 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9\": container with ID starting with 55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9 not found: ID does not exist" containerID="55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9" Oct 30 18:10:11.041873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.041813 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9"} err="failed to get container status \"55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9\": rpc error: code = NotFound desc = could not find container \"55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9\": container with ID starting with 55dd95655827f083e0c4cb4f0f12542da6e39f15188593cd384e0d5ee2144aa9 not found: ID does not exist" Oct 30 18:10:11.050319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.050298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:11.054372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054347 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-data\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054382 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-socket\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054416 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-token\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-token\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054442 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istiod-ca-cert\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054508 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-kube-api-access-m9d6t\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-kube-api-access-m9d6t\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054541 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-podinfo\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054592 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-envoy\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054631 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-data" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:11.054788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054659 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-credential-socket\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054684 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-socket" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:11.054788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054687 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-certs\") pod \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\" (UID: \"dcb2e704-471b-4172-98e8-10ec56b4f5a6\") " Oct 30 18:10:11.054996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054933 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-certs" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:11.054996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.054978 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df61b27-a947-4495-b0ae-0f345bfa9e78-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055058 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055093 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055115 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-credential-socket" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:11.055290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055269 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055362 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055394 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055432 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istiod-ca-cert" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:11.055571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055549 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztx2\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-kube-api-access-2ztx2\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055587 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055612 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.055725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055711 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.055780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055747 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.055780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055762 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.055780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055773 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.055978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.055794 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.057149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.057120 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-envoy" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:11.057222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.057172 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-podinfo" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:10:11.057319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.057299 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-token" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:11.057626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.057611 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-kube-api-access-m9d6t" pod "dcb2e704-471b-4172-98e8-10ec56b4f5a6" (UID: "dcb2e704-471b-4172-98e8-10ec56b4f5a6"). InnerVolumeSpecName "kube-api-access-m9d6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.156615 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df61b27-a947-4495-b0ae-0f345bfa9e78-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.156687 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.156721 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.156862 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.156921 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.156951 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.157038 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztx2\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-kube-api-access-2ztx2\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.157089 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.157113 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.157428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.157273 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9d6t\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-kube-api-access-m9d6t\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.158136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.157745 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.158136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.157982 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df61b27-a947-4495-b0ae-0f345bfa9e78-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.158242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.158030 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.158445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.158420 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.158528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.158460 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.160257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.158981 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.160257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.159010 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.160257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.159028 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb2e704-471b-4172-98e8-10ec56b4f5a6-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:11.161586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.161566 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.161937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.161907 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.175044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.175021 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.175130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.175089 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztx2\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-kube-api-access-2ztx2\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.185293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.185268 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:11.318792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.318761 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:11.351785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.351611 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9"] Oct 30 18:10:11.353733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.353713 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ba56e909-openshift-default-5f6986b645-6rzl9"] Oct 30 18:10:11.645990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.645957 4582 patch_prober.go:29] interesting pod/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:10:11.646170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.646016 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:10:11.661856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.661809 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr"] Oct 30 18:10:11.666706 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:10:11.666681 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df61b27_a947_4495_b0ae_0f345bfa9e78.slice/crio-598ffcb6f3ad6aab1287bea6257ebd423bbb198f0b96c4d2ddceea62fd4c0673 WatchSource:0}: Error finding container 598ffcb6f3ad6aab1287bea6257ebd423bbb198f0b96c4d2ddceea62fd4c0673: Status 404 returned error can't find the container with id 598ffcb6f3ad6aab1287bea6257ebd423bbb198f0b96c4d2ddceea62fd4c0673 Oct 30 18:10:11.669107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.669078 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:11.669203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.669141 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:11.669203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.669170 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:11.806860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:11.806812 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:12.025666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.025575 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" event={"ID":"8df61b27-a947-4495-b0ae-0f345bfa9e78","Type":"ContainerStarted","Data":"72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af"} Oct 30 18:10:12.025666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.025632 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" event={"ID":"8df61b27-a947-4495-b0ae-0f345bfa9e78","Type":"ContainerStarted","Data":"598ffcb6f3ad6aab1287bea6257ebd423bbb198f0b96c4d2ddceea62fd4c0673"} Oct 30 18:10:12.066164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.066114 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" podStartSLOduration=2.066099819 podStartE2EDuration="2.066099819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:12.064105324 +0000 UTC m=+708.888499627" watchObservedRunningTime="2026-10-30 18:10:12.066099819 +0000 UTC m=+708.890494110" Oct 30 18:10:12.319651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.319565 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:12.324797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.324772 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:12.711123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.711093 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:12.855568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.855525 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:12.950318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:12.950285 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:13.032457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:13.032389 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:13.033452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:13.033432 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:13.299229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:13.299154 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb2e704-471b-4172-98e8-10ec56b4f5a6" path="/var/lib/kubelet/pods/dcb2e704-471b-4172-98e8-10ec56b4f5a6/volumes" Oct 30 18:10:13.693075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:13.693043 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:14.869171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:14.869118 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:14.969481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:14.969439 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:14.997030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:14.997006 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:10:15.074804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.074770 4582 generic.go:356] "Generic (PLEG): container finished" podID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerID="8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8" exitCode=0 Oct 30 18:10:15.075586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.075563 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" Oct 30 18:10:15.075853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.075813 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" event={"ID":"0bae5601-08f4-4f35-8b25-e8eda5a987b1","Type":"ContainerDied","Data":"8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8"} Oct 30 18:10:15.075956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.075922 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5" event={"ID":"0bae5601-08f4-4f35-8b25-e8eda5a987b1","Type":"ContainerDied","Data":"3f022556aff7ae8fbec1e17d86903d6e34809a7c0149faacac22972dd5da8e84"} Oct 30 18:10:15.076022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.075956 4582 scope.go:122] "RemoveContainer" containerID="8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8" Oct 30 18:10:15.086682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.086665 4582 scope.go:122] "RemoveContainer" containerID="1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc" Oct 30 18:10:15.095463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.095448 4582 scope.go:122] "RemoveContainer" containerID="8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8" Oct 30 18:10:15.095753 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:10:15.095734 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8\": container with ID starting with 8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8 not found: ID does not exist" containerID="8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8" Oct 30 18:10:15.095811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.095764 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8"} err="failed to get container status \"8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8\": rpc error: code = NotFound desc = could not find container \"8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8\": container with ID starting with 8dc6f4aa0f9016de7fcbba3369a03a849bbd6b9eb1106332f43acf405a2992e8 not found: ID does not exist" Oct 30 18:10:15.095811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.095781 4582 scope.go:122] "RemoveContainer" containerID="1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc" Oct 30 18:10:15.096026 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:10:15.096005 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc\": container with ID starting with 1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc not found: ID does not exist" containerID="1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc" Oct 30 18:10:15.096122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.096030 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc"} err="failed to get container status \"1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc\": rpc error: code = NotFound desc = could not find container \"1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc\": container with ID starting with 1ecc419505663758185641004de061c740dbf4008959ab1f6bce9465c0f1eccc not found: ID does not exist" Oct 30 18:10:15.112487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112441 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-kube-api-access-gf4fm\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-kube-api-access-gf4fm\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.112613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112491 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-podinfo\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.112613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112540 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-socket\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.112613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112608 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istiod-ca-cert\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.112793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112674 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-token\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-token\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.112793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112740 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-envoy\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.112937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112811 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-credential-socket\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.112937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112918 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-certs\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.113044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.112952 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-data\") pod \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\" (UID: \"0bae5601-08f4-4f35-8b25-e8eda5a987b1\") " Oct 30 18:10:15.113410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.113386 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-socket" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:15.113410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.113396 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istiod-ca-cert" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:15.114856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.113349 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-certs" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:15.114856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.113557 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-credential-socket" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:15.114856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.113571 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-data" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:15.118322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.118286 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-podinfo" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:10:15.118322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.118301 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-token" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:15.118322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.118312 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-kube-api-access-gf4fm" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "kube-api-access-gf4fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:15.118894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.118860 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-envoy" pod "0bae5601-08f4-4f35-8b25-e8eda5a987b1" (UID: "0bae5601-08f4-4f35-8b25-e8eda5a987b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:15.214867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214809 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.214867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214866 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf4fm\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-kube-api-access-gf4fm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.215078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214883 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.215078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214897 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.215078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214910 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.215078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214922 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.215078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214934 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.215078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214946 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.215078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.214959 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bae5601-08f4-4f35-8b25-e8eda5a987b1-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.411759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.411723 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5"] Oct 30 18:10:15.420754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.420728 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-54599e13-openshift-default-d8998f799-mwtl5"] Oct 30 18:10:15.461843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:15.452690 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:16.636665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:16.636634 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:17.194051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:17.193557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:17.298975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:17.298938 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" path="/var/lib/kubelet/pods/0bae5601-08f4-4f35-8b25-e8eda5a987b1/volumes" Oct 30 18:10:17.308694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:17.308660 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:17.413255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:17.413216 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:18.667110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:18.667065 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:18.968228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:18.968146 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:19.303851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:19.303761 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:19.469357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:19.469327 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:19.574582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:19.574493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:20.887187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:20.887090 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:22.418193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:22.418154 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:22.509795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:22.509759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:22.590390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:22.590353 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:22.694239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:22.694149 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:24.131119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.131045 4582 generic.go:356] "Generic (PLEG): container finished" podID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerID="a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1" exitCode=0 Oct 30 18:10:24.131724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.131130 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" event={"ID":"03abb5af-c9ed-4ba5-b54c-04752276b68f","Type":"ContainerDied","Data":"a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1"} Oct 30 18:10:24.131724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.131630 4582 scope.go:122] "RemoveContainer" containerID="a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1" Oct 30 18:10:24.133114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.133085 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:24.133228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.133164 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:24.133228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.133214 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:24.168004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.167957 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:24.430001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.429905 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:24.678859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.677488 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6"] Oct 30 18:10:24.678859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.678220 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" Oct 30 18:10:24.678859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.678240 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" Oct 30 18:10:24.678859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.678399 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" Oct 30 18:10:24.678859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.678416 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" Oct 30 18:10:24.707522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.706061 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8"] Oct 30 18:10:24.707522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.706764 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" Oct 30 18:10:24.707522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.706782 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bae5601-08f4-4f35-8b25-e8eda5a987b1" containerName="istio-proxy" Oct 30 18:10:24.707522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.706922 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.710263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.710243 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6"] Oct 30 18:10:24.710354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.710282 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.710354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.710317 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8"] Oct 30 18:10:24.796156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.796090 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-45dc34-openshift-default-dockercfg-w86md\"" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864484 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864610 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864656 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864692 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q995b\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-kube-api-access-q995b\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864724 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864760 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864787 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864839 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864880 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spndf\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-kube-api-access-spndf\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864906 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af00f9be-3c33-41fb-a383-0664ffa85e81-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864931 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.864980 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.865037 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.865082 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.865116 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.865152 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.865176 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.872848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.865212 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966420 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966493 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966519 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966556 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966633 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966665 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966704 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966737 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q995b\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-kube-api-access-q995b\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966769 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.967635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.966806 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968343 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968401 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968454 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spndf\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-kube-api-access-spndf\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968482 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af00f9be-3c33-41fb-a383-0664ffa85e81-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968508 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968552 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968611 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.968668 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.969443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.969062 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.976385 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.976694 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.977648 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af00f9be-3c33-41fb-a383-0664ffa85e81-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.977898 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.982237 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.982483 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.982723 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.983345 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.993100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.991361 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:24.995058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.994519 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.995058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.995016 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.998840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.996912 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:24.998840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:24.997242 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:25.000371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.000325 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:25.019301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.019276 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spndf\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-kube-api-access-spndf\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:25.022717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.022692 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:25.028709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.028686 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:25.040523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.040501 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q995b\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-kube-api-access-q995b\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:25.149399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.142014 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" event={"ID":"03abb5af-c9ed-4ba5-b54c-04752276b68f","Type":"ContainerStarted","Data":"306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c"} Oct 30 18:10:25.194029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.193761 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:25.234003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.233933 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:25.234261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.234242 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:10:25.234311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.234285 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:10:25.337489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.337455 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:25.399843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.398400 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:25.410402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.410358 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6"] Oct 30 18:10:25.418252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.418214 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:25.418345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.418274 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:25.418345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.418300 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:25.804217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.804176 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8"] Oct 30 18:10:25.817745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.817707 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:25.817891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.817778 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:25.817891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:25.817805 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:26.163668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.163543 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:26.196547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.196510 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" event={"ID":"af00f9be-3c33-41fb-a383-0664ffa85e81","Type":"ContainerStarted","Data":"683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90"} Oct 30 18:10:26.196776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.196756 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" event={"ID":"af00f9be-3c33-41fb-a383-0664ffa85e81","Type":"ContainerStarted","Data":"5f77e46f0ed10678f569758fa06a54d513cc0fdf55e71166c9c978301b37887e"} Oct 30 18:10:26.216230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.216187 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" event={"ID":"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6","Type":"ContainerStarted","Data":"a976a3b054e1157602345ebbc2d4cbc903a4fed1736305b4101fd7b2c4539837"} Oct 30 18:10:26.216425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.216252 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" event={"ID":"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6","Type":"ContainerStarted","Data":"086a02701f74d86d186e71a9f37af9ec2f5f614d747383fa34920cb63a41b94d"} Oct 30 18:10:26.236452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.236413 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:10:26.236760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.236715 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:10:26.335846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.318156 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podStartSLOduration=2.318134932 podStartE2EDuration="2.318134932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:26.267339887 +0000 UTC m=+723.091734172" watchObservedRunningTime="2026-10-30 18:10:26.318134932 +0000 UTC m=+723.142529225" Oct 30 18:10:26.337964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.337924 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:26.339892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.339802 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:26.340019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:26.339890 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:10:27.041852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.040843 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:27.041852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.040989 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:10:27.041852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.041035 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 18:10:27.245652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.245223 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:10:27.245652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.245278 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:10:27.345901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.341307 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:27.345901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.341388 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:10:27.441846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.441703 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.886811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:27.886777 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:28.031223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:28.031186 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:10:28.031490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:28.031462 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 18:10:28.236363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:28.236265 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:10:28.236704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:28.236587 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:10:28.342295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:28.342258 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:28.342964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:28.342905 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:10:29.030079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:29.030042 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:10:29.030214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:29.030104 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 18:10:29.235328 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:29.233328 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:10:29.235328 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:29.233390 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:10:29.370172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:29.370058 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:29.370172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:29.370134 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:10:29.728480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:29.728324 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:30.056901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:30.034230 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:10:30.056901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:30.034306 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 18:10:30.235301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:30.235237 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:10:30.235504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:30.235308 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:10:30.378911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:30.365175 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:30.378911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:30.365234 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:10:30.541811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:30.541772 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.029599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.029261 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:10:31.029599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.029311 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 18:10:31.210737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.210629 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.239171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.238038 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:31.239171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.239036 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:31.239857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.239812 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:10:31.328915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.326364 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.328915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.327765 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podStartSLOduration=7.327749612 podStartE2EDuration="7.327749612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:26.337293194 +0000 UTC m=+723.161687487" watchObservedRunningTime="2026-10-30 18:10:31.327749612 +0000 UTC m=+728.152143907" Oct 30 18:10:31.338241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.338210 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:31.338361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.338257 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:10:31.436262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.436216 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.525864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:31.525643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:32.032048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:32.028987 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:10:32.032048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:32.029043 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 18:10:32.338878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:32.338776 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:32.338878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:32.338840 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:10:33.034873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:33.034839 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:33.035372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:33.035194 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:33.035985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:33.035966 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 18:10:33.237784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:33.237754 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.493752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:33.493420 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:33.494357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:33.494313 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:33.495347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:33.495326 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:10:34.872332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:34.872292 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:35.490016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:35.489985 4582 generic.go:356] "Generic (PLEG): container finished" podID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerID="434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f" exitCode=0 Oct 30 18:10:35.490175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:35.490057 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" event={"ID":"98b1e0ec-6439-40fe-b77b-474d327d4cc9","Type":"ContainerDied","Data":"434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f"} Oct 30 18:10:35.490586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:35.490561 4582 scope.go:122] "RemoveContainer" containerID="434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f" Oct 30 18:10:35.491733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:35.491706 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:35.491814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:35.491777 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:35.491814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:35.491809 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:36.224355 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:10:36.224319 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:10:36.497930 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:36.497845 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" event={"ID":"98b1e0ec-6439-40fe-b77b-474d327d4cc9","Type":"ContainerStarted","Data":"8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e"} Oct 30 18:10:36.588958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:36.588921 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:37.215422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:37.215388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:37.267514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:37.267464 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:37.268687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:37.268656 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:10:37.268861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:37.268704 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 18:10:38.141021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:38.140984 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:38.266336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:38.266302 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:10:38.266874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:38.266611 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 18:10:39.003120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.003092 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:39.267933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.267598 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:10:39.267933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.267653 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 18:10:39.325762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.325722 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:39.529950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.529866 4582 generic.go:356] "Generic (PLEG): container finished" podID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerID="72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af" exitCode=0 Oct 30 18:10:39.530098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.529943 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" event={"ID":"8df61b27-a947-4495-b0ae-0f345bfa9e78","Type":"ContainerDied","Data":"72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af"} Oct 30 18:10:39.530428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.530410 4582 scope.go:122] "RemoveContainer" containerID="72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af" Oct 30 18:10:39.531855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.531792 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:39.531955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.531886 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:39.531955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.531919 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:39.789033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:39.788947 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:40.267844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:40.267275 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:10:40.267844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:40.267338 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 18:10:40.537456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:40.537388 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" event={"ID":"8df61b27-a947-4495-b0ae-0f345bfa9e78","Type":"ContainerStarted","Data":"16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba"} Oct 30 18:10:41.269853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:41.269168 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:10:41.269853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:41.269227 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 18:10:41.319250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:41.319220 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:41.320298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:41.320269 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:10:41.320426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:41.320317 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:10:41.448264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:41.448232 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:42.271536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:42.271503 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:42.272498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:42.272478 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:10:42.319614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:42.319586 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:10:42.319759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:42.319631 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:10:43.032923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:43.032893 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:43.058478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:43.058434 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:43.320244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:43.320160 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:10:43.320244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:43.320212 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:10:44.319316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:44.319287 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:10:44.319483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:44.319340 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:10:44.682543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:44.682500 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:45.319435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:45.319402 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:10:45.319610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:45.319454 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:10:46.240420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:46.240385 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:46.324218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:46.324192 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:46.325040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:46.325021 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:10:46.856069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:46.856031 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:47.830447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:47.830418 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:49.447978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:49.447904 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:49.718490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:49.718411 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:50.368866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:50.368831 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:50.471140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:50.471114 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:50.574934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:50.574906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:51.172494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:51.172465 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:52.738138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:52.738106 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:54.345756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.345719 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:54.742281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.742250 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s"] Oct 30 18:10:54.745010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.744993 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.768863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.768833 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s"] Oct 30 18:10:54.790209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790158 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790236 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790372 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935d946a-ae03-4628-970a-4e0ca04d89d7-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790516 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790566 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790621 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790665 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790693 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.790836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.790737 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86sd8\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-kube-api-access-86sd8\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.891925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.891886 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935d946a-ae03-4628-970a-4e0ca04d89d7-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892005 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892052 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892103 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892144 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892175 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892335 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892218 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86sd8\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-kube-api-access-86sd8\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892395 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892452 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892535 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892577 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892734 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935d946a-ae03-4628-970a-4e0ca04d89d7-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892734 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.892802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.892770 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.894559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.894538 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.894712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.894692 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.908780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.908754 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86sd8\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-kube-api-access-86sd8\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:54.909040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:54.909022 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:55.063153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.063070 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:55.373718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.373687 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s"] Oct 30 18:10:55.378050 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:10:55.378021 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935d946a_ae03_4628_970a_4e0ca04d89d7.slice/crio-255d4cd0e7a4e1d4431bc5c068a2b288f546206ecbe4f89e5eabfc65f6dbbd53 WatchSource:0}: Error finding container 255d4cd0e7a4e1d4431bc5c068a2b288f546206ecbe4f89e5eabfc65f6dbbd53: Status 404 returned error can't find the container with id 255d4cd0e7a4e1d4431bc5c068a2b288f546206ecbe4f89e5eabfc65f6dbbd53 Oct 30 18:10:55.380644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.380611 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:55.380717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.380698 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:55.380754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.380744 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:10:55.640041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.639947 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" event={"ID":"935d946a-ae03-4628-970a-4e0ca04d89d7","Type":"ContainerStarted","Data":"8476406987a650e5daea2e43a8a99432078c566f67a6a1e0c361b3185451ad29"} Oct 30 18:10:55.640041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.640011 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" event={"ID":"935d946a-ae03-4628-970a-4e0ca04d89d7","Type":"ContainerStarted","Data":"255d4cd0e7a4e1d4431bc5c068a2b288f546206ecbe4f89e5eabfc65f6dbbd53"} Oct 30 18:10:55.680474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:55.680423 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podStartSLOduration=1.6804046910000001 podStartE2EDuration="1.680404691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:55.677481725 +0000 UTC m=+752.501876017" watchObservedRunningTime="2026-10-30 18:10:55.680404691 +0000 UTC m=+752.504798984" Oct 30 18:10:56.060906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.051300 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:56.065997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.065960 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:10:56.066336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.066315 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:10:56.066482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.066461 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:10:56.707433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.707391 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4"] Oct 30 18:10:56.711354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.711331 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.723690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.723673 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"kube-root-ca.crt\"" Oct 30 18:10:56.735965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.735942 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4"] Oct 30 18:10:56.736313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.736294 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"premium-w2-45dc34-kserve-self-signed-certs\"" Oct 30 18:10:56.736357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.736294 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"default-dockercfg-phb79\"" Oct 30 18:10:56.740531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.740502 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:56.814751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.814720 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-home\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.814925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.814774 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-model-cache\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.814925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.814905 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tmp-dir\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.815014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.814979 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-dshm\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.815014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.815008 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzc2\" (UniqueName: \"kubernetes.io/projected/8cad11d6-f33d-4ab8-8548-3e07f62429b0-kube-api-access-smzc2\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.815085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.815028 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tls-certs\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.823692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.823672 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"openshift-service-ca.crt\"" Oct 30 18:10:56.916491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.916456 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-model-cache\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.916528 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tmp-dir\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.916616 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-dshm\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.916663 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzc2\" (UniqueName: \"kubernetes.io/projected/8cad11d6-f33d-4ab8-8548-3e07f62429b0-kube-api-access-smzc2\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.916896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.916700 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tls-certs\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.916896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.916760 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-home\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.917042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.917019 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-model-cache\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.917102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.917048 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tmp-dir\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.917102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.917085 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-home\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.919412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.919388 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tls-certs\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.919767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.919745 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-dshm\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:56.932859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:56.932836 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzc2\" (UniqueName: \"kubernetes.io/projected/8cad11d6-f33d-4ab8-8548-3e07f62429b0-kube-api-access-smzc2\") pod \"premium-w2-45dc34-kserve-57d48d4b87-wt4q4\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:57.023200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.023096 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:57.067303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.067271 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:10:57.067436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.067343 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:10:57.369024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.368997 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4"] Oct 30 18:10:57.370343 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:10:57.370318 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cad11d6_f33d_4ab8_8548_3e07f62429b0.slice/crio-4aa6f38b3d2353148c6e7ebc453e5d9075f5d8d8072be265db4636201142faa3 WatchSource:0}: Error finding container 4aa6f38b3d2353148c6e7ebc453e5d9075f5d8d8072be265db4636201142faa3: Status 404 returned error can't find the container with id 4aa6f38b3d2353148c6e7ebc453e5d9075f5d8d8072be265db4636201142faa3 Oct 30 18:10:57.680723 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.680684 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" event={"ID":"8cad11d6-f33d-4ab8-8548-3e07f62429b0","Type":"ContainerStarted","Data":"fcee0131667c1b5b8db115600162216fbae06301bd754136319b522433e2ffc1"} Oct 30 18:10:57.681012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.680988 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" event={"ID":"8cad11d6-f33d-4ab8-8548-3e07f62429b0","Type":"ContainerStarted","Data":"4aa6f38b3d2353148c6e7ebc453e5d9075f5d8d8072be265db4636201142faa3"} Oct 30 18:10:57.718841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.718780 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podStartSLOduration=1.718763489 podStartE2EDuration="1.718763489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:57.717045918 +0000 UTC m=+754.541440210" watchObservedRunningTime="2026-10-30 18:10:57.718763489 +0000 UTC m=+754.543157782" Oct 30 18:10:57.771659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:57.771621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.023297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:58.023210 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:10:58.066950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:58.066910 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:10:58.067155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:58.066962 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:10:58.266878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:58.265584 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:59.067279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:59.067242 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:10:59.067681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:59.067309 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:10:59.712653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:10:59.712608 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:00.064330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:00.064234 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:00.064586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:00.064556 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:11:01.067222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:01.067186 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:01.067625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:01.067244 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:11:01.556375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:01.556339 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:02.067776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:02.067744 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:11:02.068249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:02.068157 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:11:02.068852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:02.068809 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:11:03.163422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:03.163385 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:04.728506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:04.728472 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:06.224613 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:11:06.224576 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:11:06.334834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:06.334798 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:07.023490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:07.023448 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:11:07.036432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:07.036404 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:11:07.744852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:07.744813 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:11:08.000305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:08.000208 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:09.167912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.167879 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv"] Oct 30 18:11:09.171005 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.170985 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.197687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.197660 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"openshift-service-ca.crt\"" Oct 30 18:11:09.197854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.197703 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"kube-root-ca.crt\"" Oct 30 18:11:09.197854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.197732 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"premium-w3-de7e3a-kserve-self-signed-certs\"" Oct 30 18:11:09.197854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.197766 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"default-dockercfg-p66x8\"" Oct 30 18:11:09.201707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.201686 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv"] Oct 30 18:11:09.210394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.210371 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:09.274651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.274629 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-home\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.274759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.274655 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-model-cache\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.274759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.274689 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tls-certs\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.274867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.274841 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwbm\" (UniqueName: \"kubernetes.io/projected/feccd1bd-7683-45ee-bbe1-4d20d06491d1-kube-api-access-fbwbm\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.274911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.274886 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tmp-dir\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.275004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.274992 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-dshm\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.375863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.375794 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwbm\" (UniqueName: \"kubernetes.io/projected/feccd1bd-7683-45ee-bbe1-4d20d06491d1-kube-api-access-fbwbm\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.375887 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tmp-dir\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.376022 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-dshm\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.376370 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tmp-dir\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.376414 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-home\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.376685 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-model-cache\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.376751 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tls-certs\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.376764 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-home\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.376949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.376925 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-model-cache\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.378487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.378462 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-dshm\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.379090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.379072 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tls-certs\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.392624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.392602 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwbm\" (UniqueName: \"kubernetes.io/projected/feccd1bd-7683-45ee-bbe1-4d20d06491d1-kube-api-access-fbwbm\") pod \"premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.487128 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.487051 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:09.577103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.577071 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:09.790971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:09.790938 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv"] Oct 30 18:11:09.794466 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:11:09.794424 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeccd1bd_7683_45ee_bbe1_4d20d06491d1.slice/crio-74afbd9a666f78ba6b6856374c70872d579ee9634fce67a20593e19b51eb5429 WatchSource:0}: Error finding container 74afbd9a666f78ba6b6856374c70872d579ee9634fce67a20593e19b51eb5429: Status 404 returned error can't find the container with id 74afbd9a666f78ba6b6856374c70872d579ee9634fce67a20593e19b51eb5429 Oct 30 18:11:10.752973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:10.752938 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" event={"ID":"feccd1bd-7683-45ee-bbe1-4d20d06491d1","Type":"ContainerStarted","Data":"7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c"} Oct 30 18:11:10.753400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:10.752981 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" event={"ID":"feccd1bd-7683-45ee-bbe1-4d20d06491d1","Type":"ContainerStarted","Data":"74afbd9a666f78ba6b6856374c70872d579ee9634fce67a20593e19b51eb5429"} Oct 30 18:11:10.785913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:10.785864 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podStartSLOduration=1.7858531279999998 podStartE2EDuration="1.785853128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:10.785047881 +0000 UTC m=+767.609442175" watchObservedRunningTime="2026-10-30 18:11:10.785853128 +0000 UTC m=+767.610247421" Oct 30 18:11:11.147970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.147929 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:11.359235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.359195 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7"] Oct 30 18:11:11.362388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.362367 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.386189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.386157 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7"] Oct 30 18:11:11.393893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.392000 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"kube-root-ca.crt\"" Oct 30 18:11:11.393893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.392045 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"default-dockercfg-nm4lf\"" Oct 30 18:11:11.393893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.392046 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"openshift-service-ca.crt\"" Oct 30 18:11:11.393893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.392171 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"premium-w0-921450-kserve-self-signed-certs\"" Oct 30 18:11:11.404385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.404331 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:11.487947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.487924 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:11.502458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.502426 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-model-cache\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.502519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.502494 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-home\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.502573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.502560 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tmp-dir\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.502623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.502609 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqd45\" (UniqueName: \"kubernetes.io/projected/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-kube-api-access-sqd45\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.502689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.502671 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-dshm\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.502724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.502705 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tls-certs\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.603603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.603559 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-home\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.603807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.603614 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tmp-dir\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.603807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.603668 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqd45\" (UniqueName: \"kubernetes.io/projected/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-kube-api-access-sqd45\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.603807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.603694 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-dshm\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.603807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.603720 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tls-certs\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.604054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.603878 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-model-cache\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.604091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.604046 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-home\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.604128 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.604092 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tmp-dir\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.604296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.604271 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-model-cache\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.606506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.606483 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-dshm\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.606598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.606508 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tls-certs\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.620485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.620464 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqd45\" (UniqueName: \"kubernetes.io/projected/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-kube-api-access-sqd45\") pod \"premium-w0-921450-kserve-84bdb6f7c5-7kbr7\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.679191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.679119 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:11.977068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:11.976996 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7"] Oct 30 18:11:11.981249 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:11:11.981222 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedaf5b8e_7d10_4b33_8e77_5dcc2ccf6444.slice/crio-d9071fe27565c545f1bca2b1b146e108dc61f182129b02f4108f86f926a74af3 WatchSource:0}: Error finding container d9071fe27565c545f1bca2b1b146e108dc61f182129b02f4108f86f926a74af3: Status 404 returned error can't find the container with id d9071fe27565c545f1bca2b1b146e108dc61f182129b02f4108f86f926a74af3 Oct 30 18:11:12.732338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:12.732306 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:12.768801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:12.768764 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" event={"ID":"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444","Type":"ContainerStarted","Data":"701b5d1a8d8ae8e279d30b5921be7a9f764bfb79ecc93281eb5edf8be6398ee9"} Oct 30 18:11:12.768962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:12.768813 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" event={"ID":"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444","Type":"ContainerStarted","Data":"d9071fe27565c545f1bca2b1b146e108dc61f182129b02f4108f86f926a74af3"} Oct 30 18:11:12.804782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:12.804722 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podStartSLOduration=1.804703742 podStartE2EDuration="1.804703742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:12.803001899 +0000 UTC m=+769.627396217" watchObservedRunningTime="2026-10-30 18:11:12.804703742 +0000 UTC m=+769.629098038" Oct 30 18:11:12.833358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:12.833262 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:13.679981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:13.679941 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:14.439263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:14.439235 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:16.101212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:16.101170 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:17.950409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:17.950376 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:18.460573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:18.460541 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:19.487566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:19.487535 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:19.500570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:19.500547 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:19.651670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:19.651619 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:19.820387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:19.820305 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:11:21.254637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:21.254593 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:21.680009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:21.679977 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:21.692943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:21.692919 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:21.831944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:21.831917 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:11:22.838935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:22.838906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:24.450359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:24.450325 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:26.362592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:26.362560 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:28.036194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:28.036155 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:29.615908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.615879 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:29.744446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.744405 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77"] Oct 30 18:11:29.748924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.748900 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.765517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.765492 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77"] Oct 30 18:11:29.772193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.772169 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"kube-root-ca.crt\"" Oct 30 18:11:29.772349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.772231 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"openshift-service-ca.crt\"" Oct 30 18:11:29.772349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.772258 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"default-dockercfg-jzhmp\"" Oct 30 18:11:29.772349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.772266 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"premium-w4-fe1374-kserve-self-signed-certs\"" Oct 30 18:11:29.782317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.782280 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:29.822496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.822465 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-tmp-dir\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.822622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.822527 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-home\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.822622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.822592 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f23fe3c-a468-4011-a210-aabb09c25695-tls-certs\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.822714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.822694 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-dshm\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.822874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.822859 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt45b\" (UniqueName: \"kubernetes.io/projected/7f23fe3c-a468-4011-a210-aabb09c25695-kube-api-access-xt45b\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.822922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.822890 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-model-cache\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.924413 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.924364 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-tmp-dir\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.924594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.924444 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-home\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.924594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.924469 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f23fe3c-a468-4011-a210-aabb09c25695-tls-certs\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.924594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.924509 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-dshm\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.924711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.924678 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt45b\" (UniqueName: \"kubernetes.io/projected/7f23fe3c-a468-4011-a210-aabb09c25695-kube-api-access-xt45b\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.924745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.924718 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-model-cache\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.925217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.925189 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-model-cache\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.926248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.926207 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-tmp-dir\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.926532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.926509 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-home\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.928652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.928605 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-dshm\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.932627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.932598 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f23fe3c-a468-4011-a210-aabb09c25695-tls-certs\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:29.944339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:29.944315 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt45b\" (UniqueName: \"kubernetes.io/projected/7f23fe3c-a468-4011-a210-aabb09c25695-kube-api-access-xt45b\") pod \"premium-w4-fe1374-kserve-66fd9bb5b8-psv77\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:30.061272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:30.061238 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:30.355533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:30.355455 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77"] Oct 30 18:11:30.358796 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:11:30.358764 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f23fe3c_a468_4011_a210_aabb09c25695.slice/crio-7c45736bbee7cc6da4e0dd853986a0fa68223a5d7bcc8c89e061f5b86d27712b WatchSource:0}: Error finding container 7c45736bbee7cc6da4e0dd853986a0fa68223a5d7bcc8c89e061f5b86d27712b: Status 404 returned error can't find the container with id 7c45736bbee7cc6da4e0dd853986a0fa68223a5d7bcc8c89e061f5b86d27712b Oct 30 18:11:30.895583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:30.895546 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" event={"ID":"7f23fe3c-a468-4011-a210-aabb09c25695","Type":"ContainerStarted","Data":"1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1"} Oct 30 18:11:30.896009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:30.895597 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" event={"ID":"7f23fe3c-a468-4011-a210-aabb09c25695","Type":"ContainerStarted","Data":"7c45736bbee7cc6da4e0dd853986a0fa68223a5d7bcc8c89e061f5b86d27712b"} Oct 30 18:11:30.930131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:30.930084 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podStartSLOduration=1.930069859 podStartE2EDuration="1.930069859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:30.928862102 +0000 UTC m=+787.753256396" watchObservedRunningTime="2026-10-30 18:11:30.930069859 +0000 UTC m=+787.754464150" Oct 30 18:11:31.062001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:31.061965 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:31.243530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:31.243449 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:33.087305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:33.087272 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:34.751334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:34.751303 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:36.224855 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:11:36.224799 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:11:36.352639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:36.352605 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:38.044536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:38.044490 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:39.652125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:39.652088 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:40.061862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:40.061770 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:40.074450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:40.074421 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:40.960284 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:40.960254 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:11:41.497688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:41.497653 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:43.337286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:43.337251 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:45.170445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.170412 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:45.368670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.368634 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:45.586090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.586019 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf"] Oct 30 18:11:45.589461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.589445 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.612996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.612890 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:11:45.612996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.612947 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:11:45.612996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.612951 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:11:45.612996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.612979 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:11:45.615410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.615388 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf"] Oct 30 18:11:45.699200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.699176 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.699326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.699254 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.699377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.699355 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbs7j\" (UniqueName: \"kubernetes.io/projected/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-kube-api-access-lbs7j\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.800738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.800707 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbs7j\" (UniqueName: \"kubernetes.io/projected/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-kube-api-access-lbs7j\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.800953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.800813 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.800953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.800944 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.801418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.801384 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.803193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.803171 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.817436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.817409 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbs7j\" (UniqueName: \"kubernetes.io/projected/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-kube-api-access-lbs7j\") pod \"maas-api-revoke-sub-c00d3eca-s5wvf\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:45.901664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:45.901643 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:46.192327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:46.192207 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf"] Oct 30 18:11:46.195878 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:11:46.195849 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2ccc93_6b3d_41af_a0b7_6d0f12175a3f.slice/crio-1e411bf0aef4b8a8e2bf878aa73e74e60f1bc4f6bf78efde9c1716aa5e125ecc WatchSource:0}: Error finding container 1e411bf0aef4b8a8e2bf878aa73e74e60f1bc4f6bf78efde9c1716aa5e125ecc: Status 404 returned error can't find the container with id 1e411bf0aef4b8a8e2bf878aa73e74e60f1bc4f6bf78efde9c1716aa5e125ecc Oct 30 18:11:46.834543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:46.834468 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:46.984147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:46.984111 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" event={"ID":"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f","Type":"ContainerStarted","Data":"5f2a8234f45fd6e1f04740da5c6b7b1f08aefbd853006bdcfb4224287dcb0282"} Oct 30 18:11:46.984326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:46.984181 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" event={"ID":"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f","Type":"ContainerStarted","Data":"1e411bf0aef4b8a8e2bf878aa73e74e60f1bc4f6bf78efde9c1716aa5e125ecc"} Oct 30 18:11:47.017712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:47.017655 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" podStartSLOduration=2.017639373 podStartE2EDuration="2.017639373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:47.016240796 +0000 UTC m=+803.840635098" watchObservedRunningTime="2026-10-30 18:11:47.017639373 +0000 UTC m=+803.842033659" Oct 30 18:11:47.991171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:47.991137 4582 generic.go:356] "Generic (PLEG): container finished" podID="5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" containerID="5f2a8234f45fd6e1f04740da5c6b7b1f08aefbd853006bdcfb4224287dcb0282" exitCode=0 Oct 30 18:11:47.991580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:47.991228 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" event={"ID":"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f","Type":"ContainerDied","Data":"5f2a8234f45fd6e1f04740da5c6b7b1f08aefbd853006bdcfb4224287dcb0282"} Oct 30 18:11:48.453331 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:48.453299 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:49.318595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.318276 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:49.466905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.466859 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-kube-api-access-lbs7j\" (UniqueName: \"kubernetes.io/projected/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-kube-api-access-lbs7j\") pod \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " Oct 30 18:11:49.467096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.466964 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-openshift-ca\") pod \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " Oct 30 18:11:49.467096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.467075 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-xks-ca\") pod \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\" (UID: \"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f\") " Oct 30 18:11:49.468933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.468897 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-openshift-ca" pod "5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" (UID: "5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:49.469996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.469959 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-xks-ca" pod "5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" (UID: "5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:49.473360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.473338 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-kube-api-access-lbs7j" pod "5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" (UID: "5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f"). InnerVolumeSpecName "kube-api-access-lbs7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:49.575019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.574914 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbs7j\" (UniqueName: \"kubernetes.io/projected/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-kube-api-access-lbs7j\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:11:49.575260 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.575240 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:11:49.575396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:49.575380 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:11:50.016845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:50.015320 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" event={"ID":"5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f","Type":"ContainerDied","Data":"1e411bf0aef4b8a8e2bf878aa73e74e60f1bc4f6bf78efde9c1716aa5e125ecc"} Oct 30 18:11:50.016845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:50.015386 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e411bf0aef4b8a8e2bf878aa73e74e60f1bc4f6bf78efde9c1716aa5e125ecc" Oct 30 18:11:50.016845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:50.015455 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf" Oct 30 18:11:50.305269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:50.305192 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:51.962923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:51.962895 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:53.636639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:53.636607 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:55.133575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:55.133541 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:56.737561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:56.737523 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:58.461926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:11:58.461893 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:00.085120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:00.085091 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:01.855975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:01.855941 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:03.737397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:03.737364 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:05.344422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:05.344388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:06.224240 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:12:06.224206 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:12:06.908924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.908886 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7"] Oct 30 18:12:06.909460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.909446 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" containerName="revoke-keys" Oct 30 18:12:06.909506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.909460 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" containerName="revoke-keys" Oct 30 18:12:06.909545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.909534 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" containerName="revoke-keys" Oct 30 18:12:06.911924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.911904 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:06.931309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.931281 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7"] Oct 30 18:12:06.937007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.936977 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:12:06.937129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.936976 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:06.937129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.937062 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:12:06.937129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.937081 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:12:06.982814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.982783 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195852d3-531a-4256-8eac-953a84533bc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:06.982994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.982945 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195852d3-531a-4256-8eac-953a84533bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:06.983064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:06.982996 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/195852d3-531a-4256-8eac-953a84533bc8-kube-api-access-95wq6\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.044804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.044767 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:07.084498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.084466 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195852d3-531a-4256-8eac-953a84533bc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.084664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.084534 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195852d3-531a-4256-8eac-953a84533bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.084664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.084561 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/195852d3-531a-4256-8eac-953a84533bc8-kube-api-access-95wq6\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.085256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.085233 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195852d3-531a-4256-8eac-953a84533bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.086798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.086775 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195852d3-531a-4256-8eac-953a84533bc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.101082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.101054 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/195852d3-531a-4256-8eac-953a84533bc8-kube-api-access-95wq6\") pod \"maas-api-revoke-sub-c1888862-8glm7\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.223946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.223869 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:07.519564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:07.519488 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7"] Oct 30 18:12:07.523599 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:12:07.523569 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195852d3_531a_4256_8eac_953a84533bc8.slice/crio-0c9892c4438f8024ae8d19202beb1d4117c543a39c0cb88f8fd1373ae5ef82bb WatchSource:0}: Error finding container 0c9892c4438f8024ae8d19202beb1d4117c543a39c0cb88f8fd1373ae5ef82bb: Status 404 returned error can't find the container with id 0c9892c4438f8024ae8d19202beb1d4117c543a39c0cb88f8fd1373ae5ef82bb Oct 30 18:12:08.127239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:08.127207 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" event={"ID":"195852d3-531a-4256-8eac-953a84533bc8","Type":"ContainerStarted","Data":"e7b10939366f3888dbfca90f794f6b98697c7fa7419e9586689a46f25318d9f2"} Oct 30 18:12:08.127616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:08.127256 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" event={"ID":"195852d3-531a-4256-8eac-953a84533bc8","Type":"ContainerStarted","Data":"0c9892c4438f8024ae8d19202beb1d4117c543a39c0cb88f8fd1373ae5ef82bb"} Oct 30 18:12:08.162567 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:08.162521 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" podStartSLOduration=2.162504612 podStartE2EDuration="2.162504612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:08.159332249 +0000 UTC m=+824.983726551" watchObservedRunningTime="2026-10-30 18:12:08.162504612 +0000 UTC m=+824.986898906" Oct 30 18:12:08.628590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:08.628559 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:09.134896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:09.134865 4582 generic.go:356] "Generic (PLEG): container finished" podID="195852d3-531a-4256-8eac-953a84533bc8" containerID="e7b10939366f3888dbfca90f794f6b98697c7fa7419e9586689a46f25318d9f2" exitCode=0 Oct 30 18:12:09.135353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:09.134948 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" event={"ID":"195852d3-531a-4256-8eac-953a84533bc8","Type":"ContainerDied","Data":"e7b10939366f3888dbfca90f794f6b98697c7fa7419e9586689a46f25318d9f2"} Oct 30 18:12:10.290770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.290742 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:10.324928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.324896 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195852d3-531a-4256-8eac-953a84533bc8-kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/195852d3-531a-4256-8eac-953a84533bc8-kube-api-access-95wq6\") pod \"195852d3-531a-4256-8eac-953a84533bc8\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " Oct 30 18:12:10.325189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.324974 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195852d3-531a-4256-8eac-953a84533bc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195852d3-531a-4256-8eac-953a84533bc8-maas-api-xks-ca\") pod \"195852d3-531a-4256-8eac-953a84533bc8\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " Oct 30 18:12:10.325189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.325145 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195852d3-531a-4256-8eac-953a84533bc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195852d3-531a-4256-8eac-953a84533bc8-maas-api-openshift-ca\") pod \"195852d3-531a-4256-8eac-953a84533bc8\" (UID: \"195852d3-531a-4256-8eac-953a84533bc8\") " Oct 30 18:12:10.325742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.325718 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195852d3-531a-4256-8eac-953a84533bc8-maas-api-openshift-ca" pod "195852d3-531a-4256-8eac-953a84533bc8" (UID: "195852d3-531a-4256-8eac-953a84533bc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:10.327362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.327330 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195852d3-531a-4256-8eac-953a84533bc8-maas-api-xks-ca" pod "195852d3-531a-4256-8eac-953a84533bc8" (UID: "195852d3-531a-4256-8eac-953a84533bc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:10.327477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.327412 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195852d3-531a-4256-8eac-953a84533bc8-kube-api-access-95wq6" pod "195852d3-531a-4256-8eac-953a84533bc8" (UID: "195852d3-531a-4256-8eac-953a84533bc8"). InnerVolumeSpecName "kube-api-access-95wq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:10.370831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.370790 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:10.427279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.427212 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/195852d3-531a-4256-8eac-953a84533bc8-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:10.427279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.427235 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wq6\" (UniqueName: \"kubernetes.io/projected/195852d3-531a-4256-8eac-953a84533bc8-kube-api-access-95wq6\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:10.427279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.427246 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/195852d3-531a-4256-8eac-953a84533bc8-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:10.762288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.762216 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s"] Oct 30 18:12:10.762758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.762745 4582 cpu_manager.go:414] "containerMap: removing container" podUID="195852d3-531a-4256-8eac-953a84533bc8" containerName="revoke-keys" Oct 30 18:12:10.762799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.762759 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195852d3-531a-4256-8eac-953a84533bc8" containerName="revoke-keys" Oct 30 18:12:10.762877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.762865 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="195852d3-531a-4256-8eac-953a84533bc8" containerName="revoke-keys" Oct 30 18:12:10.766142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.766124 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.783402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.783377 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s"] Oct 30 18:12:10.832801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.832770 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.832959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.832907 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.832959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.832955 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnmd\" (UniqueName: \"kubernetes.io/projected/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-kube-api-access-8xnmd\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.934518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.934483 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.934681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.934540 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnmd\" (UniqueName: \"kubernetes.io/projected/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-kube-api-access-8xnmd\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.934681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.934609 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.935242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.935223 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.936794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.936776 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:10.951622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:10.951601 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnmd\" (UniqueName: \"kubernetes.io/projected/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-kube-api-access-8xnmd\") pod \"maas-api-revoke-sub-1c1bd91c-jkl2s\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:11.079723 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:11.079653 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:11.152178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:11.152150 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" Oct 30 18:12:11.152325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:11.152155 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7" event={"ID":"195852d3-531a-4256-8eac-953a84533bc8","Type":"ContainerDied","Data":"0c9892c4438f8024ae8d19202beb1d4117c543a39c0cb88f8fd1373ae5ef82bb"} Oct 30 18:12:11.152325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:11.152285 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9892c4438f8024ae8d19202beb1d4117c543a39c0cb88f8fd1373ae5ef82bb" Oct 30 18:12:11.377292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:11.377267 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s"] Oct 30 18:12:11.379401 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:12:11.379375 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a07f3a1_972c_4f92_bcf0_1f38632b9f17.slice/crio-15767855db198dfb9eb204bd7200fd8883aa8ec0be9a966f348d2f3f75a93d2f WatchSource:0}: Error finding container 15767855db198dfb9eb204bd7200fd8883aa8ec0be9a966f348d2f3f75a93d2f: Status 404 returned error can't find the container with id 15767855db198dfb9eb204bd7200fd8883aa8ec0be9a966f348d2f3f75a93d2f Oct 30 18:12:12.160036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:12.159999 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" event={"ID":"3a07f3a1-972c-4f92-bcf0-1f38632b9f17","Type":"ContainerStarted","Data":"3e02a0f500fb2e51ae05ed60f912b709387d5a7affbb4c1711c1e7a8f112f774"} Oct 30 18:12:12.160206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:12.160057 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" event={"ID":"3a07f3a1-972c-4f92-bcf0-1f38632b9f17","Type":"ContainerStarted","Data":"15767855db198dfb9eb204bd7200fd8883aa8ec0be9a966f348d2f3f75a93d2f"} Oct 30 18:12:12.187608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:12.187577 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:12.195389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:12.195344 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" podStartSLOduration=2.195332186 podStartE2EDuration="2.195332186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:12.193299072 +0000 UTC m=+829.017693361" watchObservedRunningTime="2026-10-30 18:12:12.195332186 +0000 UTC m=+829.019726477" Oct 30 18:12:12.911117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:12.911087 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:13.168478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:13.168377 4582 generic.go:356] "Generic (PLEG): container finished" podID="3a07f3a1-972c-4f92-bcf0-1f38632b9f17" containerID="3e02a0f500fb2e51ae05ed60f912b709387d5a7affbb4c1711c1e7a8f112f774" exitCode=0 Oct 30 18:12:13.168478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:13.168455 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" event={"ID":"3a07f3a1-972c-4f92-bcf0-1f38632b9f17","Type":"ContainerDied","Data":"3e02a0f500fb2e51ae05ed60f912b709387d5a7affbb4c1711c1e7a8f112f774"} Oct 30 18:12:13.876378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:13.876345 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:14.319526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.319503 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:14.371064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.371032 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-xks-ca\") pod \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " Oct 30 18:12:14.371234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.371186 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-kube-api-access-8xnmd\" (UniqueName: \"kubernetes.io/projected/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-kube-api-access-8xnmd\") pod \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " Oct 30 18:12:14.371343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.371326 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-openshift-ca\") pod \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\" (UID: \"3a07f3a1-972c-4f92-bcf0-1f38632b9f17\") " Oct 30 18:12:14.371694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.371667 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-openshift-ca" pod "3a07f3a1-972c-4f92-bcf0-1f38632b9f17" (UID: "3a07f3a1-972c-4f92-bcf0-1f38632b9f17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:14.371934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.371917 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:14.373487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.373449 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-xks-ca" pod "3a07f3a1-972c-4f92-bcf0-1f38632b9f17" (UID: "3a07f3a1-972c-4f92-bcf0-1f38632b9f17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:14.373571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.373478 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-kube-api-access-8xnmd" pod "3a07f3a1-972c-4f92-bcf0-1f38632b9f17" (UID: "3a07f3a1-972c-4f92-bcf0-1f38632b9f17"). InnerVolumeSpecName "kube-api-access-8xnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:14.473569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.473493 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xnmd\" (UniqueName: \"kubernetes.io/projected/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-kube-api-access-8xnmd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:14.473569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:14.473525 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07f3a1-972c-4f92-bcf0-1f38632b9f17-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:15.182090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.182059 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" Oct 30 18:12:15.182280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.182089 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s" event={"ID":"3a07f3a1-972c-4f92-bcf0-1f38632b9f17","Type":"ContainerDied","Data":"15767855db198dfb9eb204bd7200fd8883aa8ec0be9a966f348d2f3f75a93d2f"} Oct 30 18:12:15.182280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.182156 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15767855db198dfb9eb204bd7200fd8883aa8ec0be9a966f348d2f3f75a93d2f" Oct 30 18:12:15.445742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.445663 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:15.873227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.873198 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s"] Oct 30 18:12:15.873704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.873692 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3a07f3a1-972c-4f92-bcf0-1f38632b9f17" containerName="revoke-keys" Oct 30 18:12:15.873759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.873706 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a07f3a1-972c-4f92-bcf0-1f38632b9f17" containerName="revoke-keys" Oct 30 18:12:15.873849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.873839 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a07f3a1-972c-4f92-bcf0-1f38632b9f17" containerName="revoke-keys" Oct 30 18:12:15.877219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.877202 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:15.895795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.895764 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s"] Oct 30 18:12:15.901606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.901578 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:12:15.901805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.901578 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:12:15.901805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.901618 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:12:15.901805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.901665 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:15.990396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.990361 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:15.990604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.990432 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:15.990604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:15.990484 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/26455faa-a6b9-4845-9d46-4ba0ff02c383-kube-api-access-x9mq6\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.091779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.091735 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.092011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.091885 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.092011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.091969 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/26455faa-a6b9-4845-9d46-4ba0ff02c383-kube-api-access-x9mq6\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.092571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.092544 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.094879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.094844 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.108996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.108966 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/26455faa-a6b9-4845-9d46-4ba0ff02c383-kube-api-access-x9mq6\") pod \"maas-api-revoke-sub-273e86f3-nf26s\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.190758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.190649 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:16.629568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:16.628744 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s"] Oct 30 18:12:16.631446 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:12:16.631408 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26455faa_a6b9_4845_9d46_4ba0ff02c383.slice/crio-474be65f8304ec108959380d6891dc72295edf92d54cefc4f03069f61f17ea81 WatchSource:0}: Error finding container 474be65f8304ec108959380d6891dc72295edf92d54cefc4f03069f61f17ea81: Status 404 returned error can't find the container with id 474be65f8304ec108959380d6891dc72295edf92d54cefc4f03069f61f17ea81 Oct 30 18:12:17.198728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:17.198696 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" event={"ID":"26455faa-a6b9-4845-9d46-4ba0ff02c383","Type":"ContainerStarted","Data":"80b7ce92357b5997873777f3b3c1854a128dcb70b40bd8d0b48b253bb4b0dff9"} Oct 30 18:12:17.198895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:17.198744 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" event={"ID":"26455faa-a6b9-4845-9d46-4ba0ff02c383","Type":"ContainerStarted","Data":"474be65f8304ec108959380d6891dc72295edf92d54cefc4f03069f61f17ea81"} Oct 30 18:12:17.233551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:17.233436 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" podStartSLOduration=2.233421319 podStartE2EDuration="2.233421319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:17.231233726 +0000 UTC m=+834.055628028" watchObservedRunningTime="2026-10-30 18:12:17.233421319 +0000 UTC m=+834.057815611" Oct 30 18:12:17.348375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:17.348343 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:18.205973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:18.205939 4582 generic.go:356] "Generic (PLEG): container finished" podID="26455faa-a6b9-4845-9d46-4ba0ff02c383" containerID="80b7ce92357b5997873777f3b3c1854a128dcb70b40bd8d0b48b253bb4b0dff9" exitCode=0 Oct 30 18:12:18.206374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:18.206026 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" event={"ID":"26455faa-a6b9-4845-9d46-4ba0ff02c383","Type":"ContainerDied","Data":"80b7ce92357b5997873777f3b3c1854a128dcb70b40bd8d0b48b253bb4b0dff9"} Oct 30 18:12:19.074419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.074388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:19.345422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.345401 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:19.426333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.426301 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-openshift-ca\") pod \"26455faa-a6b9-4845-9d46-4ba0ff02c383\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " Oct 30 18:12:19.426497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.426482 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26455faa-a6b9-4845-9d46-4ba0ff02c383-kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/26455faa-a6b9-4845-9d46-4ba0ff02c383-kube-api-access-x9mq6\") pod \"26455faa-a6b9-4845-9d46-4ba0ff02c383\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " Oct 30 18:12:19.426616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.426594 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-xks-ca\") pod \"26455faa-a6b9-4845-9d46-4ba0ff02c383\" (UID: \"26455faa-a6b9-4845-9d46-4ba0ff02c383\") " Oct 30 18:12:19.426616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.426605 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-openshift-ca" pod "26455faa-a6b9-4845-9d46-4ba0ff02c383" (UID: "26455faa-a6b9-4845-9d46-4ba0ff02c383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:19.427012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.426996 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:19.428498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.428480 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-xks-ca" pod "26455faa-a6b9-4845-9d46-4ba0ff02c383" (UID: "26455faa-a6b9-4845-9d46-4ba0ff02c383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:19.428590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.428559 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26455faa-a6b9-4845-9d46-4ba0ff02c383-kube-api-access-x9mq6" pod "26455faa-a6b9-4845-9d46-4ba0ff02c383" (UID: "26455faa-a6b9-4845-9d46-4ba0ff02c383"). InnerVolumeSpecName "kube-api-access-x9mq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:19.527732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.527707 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26455faa-a6b9-4845-9d46-4ba0ff02c383-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:19.527878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:19.527737 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/26455faa-a6b9-4845-9d46-4ba0ff02c383-kube-api-access-x9mq6\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:20.218700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:20.218671 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" Oct 30 18:12:20.218700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:20.218688 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s" event={"ID":"26455faa-a6b9-4845-9d46-4ba0ff02c383","Type":"ContainerDied","Data":"474be65f8304ec108959380d6891dc72295edf92d54cefc4f03069f61f17ea81"} Oct 30 18:12:20.218925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:20.218730 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474be65f8304ec108959380d6891dc72295edf92d54cefc4f03069f61f17ea81" Oct 30 18:12:20.770836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:20.770794 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:21.362921 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.362890 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt"] Oct 30 18:12:21.363463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.363445 4582 cpu_manager.go:414] "containerMap: removing container" podUID="26455faa-a6b9-4845-9d46-4ba0ff02c383" containerName="revoke-keys" Oct 30 18:12:21.363576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.363465 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26455faa-a6b9-4845-9d46-4ba0ff02c383" containerName="revoke-keys" Oct 30 18:12:21.363630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.363579 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="26455faa-a6b9-4845-9d46-4ba0ff02c383" containerName="revoke-keys" Oct 30 18:12:21.367038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.367016 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.380262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.380241 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:21.380449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.380428 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:12:21.389782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.389760 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:12:21.389961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.389942 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:12:21.394295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.394270 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt"] Oct 30 18:12:21.448831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.448795 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.448955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.448926 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.449048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.449033 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbm6\" (UniqueName: \"kubernetes.io/projected/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-kube-api-access-4sbm6\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.549852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.549785 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.550038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.549922 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbm6\" (UniqueName: \"kubernetes.io/projected/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-kube-api-access-4sbm6\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.550038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.549994 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.550656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.550632 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.552120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.552097 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.570244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.570221 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbm6\" (UniqueName: \"kubernetes.io/projected/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-kube-api-access-4sbm6\") pod \"maas-api-revoke-sub-0f2da5c0-2n5rt\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.680115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.680084 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:21.977789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:21.977717 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt"] Oct 30 18:12:21.981248 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:12:21.981221 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bc93f7_3ad3_4813_9fd1_ecb8a3045726.slice/crio-5fdd57f1818cbe5f955d66e5fe48cff95c80806cd7bf7705dbeb99e214aa04b4 WatchSource:0}: Error finding container 5fdd57f1818cbe5f955d66e5fe48cff95c80806cd7bf7705dbeb99e214aa04b4: Status 404 returned error can't find the container with id 5fdd57f1818cbe5f955d66e5fe48cff95c80806cd7bf7705dbeb99e214aa04b4 Oct 30 18:12:22.233011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:22.232929 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" event={"ID":"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726","Type":"ContainerStarted","Data":"5fdd57f1818cbe5f955d66e5fe48cff95c80806cd7bf7705dbeb99e214aa04b4"} Oct 30 18:12:22.648660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:22.648623 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:23.240152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:23.240116 4582 generic.go:356] "Generic (PLEG): container finished" podID="e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" containerID="b3188e559adfc59d1ce9c64cabc08430aef499d52cedda2414e7c35006b34080" exitCode=0 Oct 30 18:12:23.240539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:23.240196 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" event={"ID":"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726","Type":"ContainerDied","Data":"b3188e559adfc59d1ce9c64cabc08430aef499d52cedda2414e7c35006b34080"} Oct 30 18:12:24.338639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.338598 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:24.420687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.420385 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:24.489743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.489707 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-xks-ca\") pod \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " Oct 30 18:12:24.489928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.489792 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-kube-api-access-4sbm6\" (UniqueName: \"kubernetes.io/projected/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-kube-api-access-4sbm6\") pod \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " Oct 30 18:12:24.489928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.489913 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-openshift-ca\") pod \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\" (UID: \"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726\") " Oct 30 18:12:24.490363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.490327 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-openshift-ca" pod "e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" (UID: "e8bc93f7-3ad3-4813-9fd1-ecb8a3045726"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:24.491961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.491925 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-xks-ca" pod "e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" (UID: "e8bc93f7-3ad3-4813-9fd1-ecb8a3045726"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:24.492082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.491992 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-kube-api-access-4sbm6" pod "e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" (UID: "e8bc93f7-3ad3-4813-9fd1-ecb8a3045726"). InnerVolumeSpecName "kube-api-access-4sbm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:24.591186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.591161 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:24.591186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.591183 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:24.591334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:24.591195 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sbm6\" (UniqueName: \"kubernetes.io/projected/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726-kube-api-access-4sbm6\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:25.260943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:25.260911 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" Oct 30 18:12:25.261094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:25.260914 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt" event={"ID":"e8bc93f7-3ad3-4813-9fd1-ecb8a3045726","Type":"ContainerDied","Data":"5fdd57f1818cbe5f955d66e5fe48cff95c80806cd7bf7705dbeb99e214aa04b4"} Oct 30 18:12:25.261094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:25.261068 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fdd57f1818cbe5f955d66e5fe48cff95c80806cd7bf7705dbeb99e214aa04b4" Oct 30 18:12:25.953440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:25.953407 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:27.438713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:27.438681 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:29.143573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.143538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:29.276575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.276544 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c"] Oct 30 18:12:29.277100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.277086 4582 cpu_manager.go:414] "containerMap: removing container" podUID="e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" containerName="revoke-keys" Oct 30 18:12:29.277164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.277102 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" containerName="revoke-keys" Oct 30 18:12:29.277203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.277193 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" containerName="revoke-keys" Oct 30 18:12:29.280712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.280692 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.293889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.293863 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:29.301177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.301159 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c"] Oct 30 18:12:29.303457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.303437 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:12:29.303573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.303491 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:12:29.303633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.303570 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:12:29.340323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.340298 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.340423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.340363 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.340548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.340529 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbds\" (UniqueName: \"kubernetes.io/projected/29c51ec2-8510-46fb-927c-2723e914f06c-kube-api-access-dbbds\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.441588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.441518 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.441588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.441572 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.441732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.441604 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbds\" (UniqueName: \"kubernetes.io/projected/29c51ec2-8510-46fb-927c-2723e914f06c-kube-api-access-dbbds\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.442308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.442281 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.443762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.443734 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.457988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.457965 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbds\" (UniqueName: \"kubernetes.io/projected/29c51ec2-8510-46fb-927c-2723e914f06c-kube-api-access-dbbds\") pod \"maas-api-revoke-sub-95cbd13b-b4c5c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.593948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.593910 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:29.893393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:29.893363 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c"] Oct 30 18:12:29.896844 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:12:29.896800 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c51ec2_8510_46fb_927c_2723e914f06c.slice/crio-094b01eb15d90abe5fa931bf9305667a648b10624a4be081ff5eab232fe4a48a WatchSource:0}: Error finding container 094b01eb15d90abe5fa931bf9305667a648b10624a4be081ff5eab232fe4a48a: Status 404 returned error can't find the container with id 094b01eb15d90abe5fa931bf9305667a648b10624a4be081ff5eab232fe4a48a Oct 30 18:12:30.290954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:30.290916 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" event={"ID":"29c51ec2-8510-46fb-927c-2723e914f06c","Type":"ContainerStarted","Data":"d61caefdfc0e9a5d7d2bbf56732484cc6be03f773f5e0d3ad9a6cdf2b021e7b8"} Oct 30 18:12:30.291383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:30.290971 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" event={"ID":"29c51ec2-8510-46fb-927c-2723e914f06c","Type":"ContainerStarted","Data":"094b01eb15d90abe5fa931bf9305667a648b10624a4be081ff5eab232fe4a48a"} Oct 30 18:12:30.323851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:30.323786 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" podStartSLOduration=1.323773594 podStartE2EDuration="1.323773594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:30.322720562 +0000 UTC m=+847.147114855" watchObservedRunningTime="2026-10-30 18:12:30.323773594 +0000 UTC m=+847.148167886" Oct 30 18:12:30.722942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:30.722915 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:31.298989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:31.298956 4582 generic.go:356] "Generic (PLEG): container finished" podID="29c51ec2-8510-46fb-927c-2723e914f06c" containerID="d61caefdfc0e9a5d7d2bbf56732484cc6be03f773f5e0d3ad9a6cdf2b021e7b8" exitCode=0 Oct 30 18:12:31.299997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:31.299976 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" event={"ID":"29c51ec2-8510-46fb-927c-2723e914f06c","Type":"ContainerDied","Data":"d61caefdfc0e9a5d7d2bbf56732484cc6be03f773f5e0d3ad9a6cdf2b021e7b8"} Oct 30 18:12:32.346540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.346507 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:32.458596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.458572 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:32.575757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.575684 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-openshift-ca\") pod \"29c51ec2-8510-46fb-927c-2723e914f06c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " Oct 30 18:12:32.575757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.575745 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-xks-ca\") pod \"29c51ec2-8510-46fb-927c-2723e914f06c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " Oct 30 18:12:32.575978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.575927 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c51ec2-8510-46fb-927c-2723e914f06c-kube-api-access-dbbds\" (UniqueName: \"kubernetes.io/projected/29c51ec2-8510-46fb-927c-2723e914f06c-kube-api-access-dbbds\") pod \"29c51ec2-8510-46fb-927c-2723e914f06c\" (UID: \"29c51ec2-8510-46fb-927c-2723e914f06c\") " Oct 30 18:12:32.576076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.576054 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-openshift-ca" pod "29c51ec2-8510-46fb-927c-2723e914f06c" (UID: "29c51ec2-8510-46fb-927c-2723e914f06c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:32.576234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.576219 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:32.577718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.577698 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-xks-ca" pod "29c51ec2-8510-46fb-927c-2723e914f06c" (UID: "29c51ec2-8510-46fb-927c-2723e914f06c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:32.577857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.577811 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c51ec2-8510-46fb-927c-2723e914f06c-kube-api-access-dbbds" pod "29c51ec2-8510-46fb-927c-2723e914f06c" (UID: "29c51ec2-8510-46fb-927c-2723e914f06c"). InnerVolumeSpecName "kube-api-access-dbbds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:32.677390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.677358 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbds\" (UniqueName: \"kubernetes.io/projected/29c51ec2-8510-46fb-927c-2723e914f06c-kube-api-access-dbbds\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:32.677390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:32.677384 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c51ec2-8510-46fb-927c-2723e914f06c-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:33.314745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:33.314715 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" Oct 30 18:12:33.314906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:33.314744 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c" event={"ID":"29c51ec2-8510-46fb-927c-2723e914f06c","Type":"ContainerDied","Data":"094b01eb15d90abe5fa931bf9305667a648b10624a4be081ff5eab232fe4a48a"} Oct 30 18:12:33.314906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:33.314803 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094b01eb15d90abe5fa931bf9305667a648b10624a4be081ff5eab232fe4a48a" Oct 30 18:12:33.966144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:33.966107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:34.608310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:34.608270 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:35.553160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:35.553126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:36.224967 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:12:36.224931 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:12:37.100007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:37.099973 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:38.855043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:38.855012 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:40.973207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:40.973173 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:42.651951 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:42.651921 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:44.257955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:44.257927 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:45.848346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:45.848313 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:47.343314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:47.343283 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:49.040661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:49.040627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:50.746235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:50.746202 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:51.470074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:51.470041 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:52.179150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.179116 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n"] Oct 30 18:12:52.179626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.179613 4582 cpu_manager.go:414] "containerMap: removing container" podUID="29c51ec2-8510-46fb-927c-2723e914f06c" containerName="revoke-keys" Oct 30 18:12:52.179676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.179627 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c51ec2-8510-46fb-927c-2723e914f06c" containerName="revoke-keys" Oct 30 18:12:52.179736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.179726 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c51ec2-8510-46fb-927c-2723e914f06c" containerName="revoke-keys" Oct 30 18:12:52.184506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.184489 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.198347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.198319 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:12:52.198454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.198356 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:12:52.205282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.205236 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n"] Oct 30 18:12:52.208070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.208047 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:52.208199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.208182 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:12:52.220310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.220281 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.220455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.220322 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.220563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.220538 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9fb\" (UniqueName: \"kubernetes.io/projected/be7f69e1-a92d-4dce-a96c-e03576d7a20b-kube-api-access-5s9fb\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.321155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.321121 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.321351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.321165 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.321351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.321245 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9fb\" (UniqueName: \"kubernetes.io/projected/be7f69e1-a92d-4dce-a96c-e03576d7a20b-kube-api-access-5s9fb\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.321844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.321803 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.323582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.323553 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.341301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.341274 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9fb\" (UniqueName: \"kubernetes.io/projected/be7f69e1-a92d-4dce-a96c-e03576d7a20b-kube-api-access-5s9fb\") pod \"maas-api-revoke-sub-966dc504-pb66n\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.436079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.436002 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:52.498100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.498061 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:52.802269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:52.802188 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n"] Oct 30 18:12:52.805956 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:12:52.805925 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7f69e1_a92d_4dce_a96c_e03576d7a20b.slice/crio-fc0014226476f7b716757bc55410b555870dc3c42ce98d1c23e6b46b447dc7e9 WatchSource:0}: Error finding container fc0014226476f7b716757bc55410b555870dc3c42ce98d1c23e6b46b447dc7e9: Status 404 returned error can't find the container with id fc0014226476f7b716757bc55410b555870dc3c42ce98d1c23e6b46b447dc7e9 Oct 30 18:12:53.437159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:53.437126 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" event={"ID":"be7f69e1-a92d-4dce-a96c-e03576d7a20b","Type":"ContainerStarted","Data":"2833bf406ff2a9b1d1ed4a1d9533e4712b99f780853accb87f74382fbd173de6"} Oct 30 18:12:53.437563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:53.437176 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" event={"ID":"be7f69e1-a92d-4dce-a96c-e03576d7a20b","Type":"ContainerStarted","Data":"fc0014226476f7b716757bc55410b555870dc3c42ce98d1c23e6b46b447dc7e9"} Oct 30 18:12:53.471926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:53.471865 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" podStartSLOduration=1.471847825 podStartE2EDuration="1.471847825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:53.469515315 +0000 UTC m=+870.293909606" watchObservedRunningTime="2026-10-30 18:12:53.471847825 +0000 UTC m=+870.296242132" Oct 30 18:12:54.033288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:54.033251 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:54.444932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:54.444891 4582 generic.go:356] "Generic (PLEG): container finished" podID="be7f69e1-a92d-4dce-a96c-e03576d7a20b" containerID="2833bf406ff2a9b1d1ed4a1d9533e4712b99f780853accb87f74382fbd173de6" exitCode=0 Oct 30 18:12:54.445304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:54.444946 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" event={"ID":"be7f69e1-a92d-4dce-a96c-e03576d7a20b","Type":"ContainerDied","Data":"2833bf406ff2a9b1d1ed4a1d9533e4712b99f780853accb87f74382fbd173de6"} Oct 30 18:12:55.595292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.595265 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:55.653501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.653466 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-xks-ca\") pod \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " Oct 30 18:12:55.653668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.653537 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-openshift-ca\") pod \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " Oct 30 18:12:55.653668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.653556 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7f69e1-a92d-4dce-a96c-e03576d7a20b-kube-api-access-5s9fb\" (UniqueName: \"kubernetes.io/projected/be7f69e1-a92d-4dce-a96c-e03576d7a20b-kube-api-access-5s9fb\") pod \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\" (UID: \"be7f69e1-a92d-4dce-a96c-e03576d7a20b\") " Oct 30 18:12:55.653931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.653898 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-openshift-ca" pod "be7f69e1-a92d-4dce-a96c-e03576d7a20b" (UID: "be7f69e1-a92d-4dce-a96c-e03576d7a20b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:55.655764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.655729 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7f69e1-a92d-4dce-a96c-e03576d7a20b-kube-api-access-5s9fb" pod "be7f69e1-a92d-4dce-a96c-e03576d7a20b" (UID: "be7f69e1-a92d-4dce-a96c-e03576d7a20b"). InnerVolumeSpecName "kube-api-access-5s9fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:55.655963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.655770 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-xks-ca" pod "be7f69e1-a92d-4dce-a96c-e03576d7a20b" (UID: "be7f69e1-a92d-4dce-a96c-e03576d7a20b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:55.730707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.730626 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:55.754879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.754849 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:55.754879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.754877 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7f69e1-a92d-4dce-a96c-e03576d7a20b-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:55.754879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:55.754888 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s9fb\" (UniqueName: \"kubernetes.io/projected/be7f69e1-a92d-4dce-a96c-e03576d7a20b-kube-api-access-5s9fb\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:12:56.458408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:56.458377 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" Oct 30 18:12:56.458572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:56.458412 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n" event={"ID":"be7f69e1-a92d-4dce-a96c-e03576d7a20b","Type":"ContainerDied","Data":"fc0014226476f7b716757bc55410b555870dc3c42ce98d1c23e6b46b447dc7e9"} Oct 30 18:12:56.458572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:56.458460 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0014226476f7b716757bc55410b555870dc3c42ce98d1c23e6b46b447dc7e9" Oct 30 18:12:57.384025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:57.383981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:59.158768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:12:59.158732 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:00.767762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:00.767720 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:02.466017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.465984 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:02.530101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.530062 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx"] Oct 30 18:13:02.530891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.530855 4582 cpu_manager.go:414] "containerMap: removing container" podUID="be7f69e1-a92d-4dce-a96c-e03576d7a20b" containerName="revoke-keys" Oct 30 18:13:02.530891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.530891 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7f69e1-a92d-4dce-a96c-e03576d7a20b" containerName="revoke-keys" Oct 30 18:13:02.531233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.531031 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7f69e1-a92d-4dce-a96c-e03576d7a20b" containerName="revoke-keys" Oct 30 18:13:02.534801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.534783 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.543771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.543748 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscms\" (UniqueName: \"kubernetes.io/projected/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-kube-api-access-nscms\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.543894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.543844 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.543983 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.543963 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.548462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.548437 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:13:02.548462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.548447 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:02.548647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.548541 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:13:02.552755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.552732 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx"] Oct 30 18:13:02.557457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.557433 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:13:02.645583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.645550 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscms\" (UniqueName: \"kubernetes.io/projected/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-kube-api-access-nscms\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.645738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.645608 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.645738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.645659 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.646262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.646236 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.648027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.648006 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.661451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.661427 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscms\" (UniqueName: \"kubernetes.io/projected/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-kube-api-access-nscms\") pod \"maas-api-revoke-sub-f16f5427-cdncx\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:02.849681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:02.849605 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:03.144361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:03.144338 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx"] Oct 30 18:13:03.147280 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:13:03.147240 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f4c4cf_2eb2_4490_8719_795ea7f3ff53.slice/crio-f1724b0753e6483eb32e53d51c2b565cbe7adf6625f41eb3e6000c24368b6d66 WatchSource:0}: Error finding container f1724b0753e6483eb32e53d51c2b565cbe7adf6625f41eb3e6000c24368b6d66: Status 404 returned error can't find the container with id f1724b0753e6483eb32e53d51c2b565cbe7adf6625f41eb3e6000c24368b6d66 Oct 30 18:13:03.507006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:03.506979 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" event={"ID":"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53","Type":"ContainerStarted","Data":"24ca5d59fb0db3771e6f1bffb45175a0183d74b361159a7b44654cf1b1a8dab2"} Oct 30 18:13:03.507416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:03.507023 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" event={"ID":"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53","Type":"ContainerStarted","Data":"f1724b0753e6483eb32e53d51c2b565cbe7adf6625f41eb3e6000c24368b6d66"} Oct 30 18:13:03.544600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:03.544552 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" podStartSLOduration=1.544537242 podStartE2EDuration="1.544537242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:03.542272758 +0000 UTC m=+880.366667060" watchObservedRunningTime="2026-10-30 18:13:03.544537242 +0000 UTC m=+880.368931534" Oct 30 18:13:04.112372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:04.112342 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:04.513986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:04.513955 4582 generic.go:356] "Generic (PLEG): container finished" podID="c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" containerID="24ca5d59fb0db3771e6f1bffb45175a0183d74b361159a7b44654cf1b1a8dab2" exitCode=0 Oct 30 18:13:04.514378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:04.514043 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" event={"ID":"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53","Type":"ContainerDied","Data":"24ca5d59fb0db3771e6f1bffb45175a0183d74b361159a7b44654cf1b1a8dab2"} Oct 30 18:13:05.645750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.645713 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:05.675812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.675786 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:05.781523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.781399 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-xks-ca\") pod \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " Oct 30 18:13:05.781523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.781468 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-openshift-ca\") pod \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " Oct 30 18:13:05.781814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.781553 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-kube-api-access-nscms\" (UniqueName: \"kubernetes.io/projected/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-kube-api-access-nscms\") pod \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\" (UID: \"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53\") " Oct 30 18:13:05.781910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.781885 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-openshift-ca" pod "c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" (UID: "c8f4c4cf-2eb2-4490-8719-795ea7f3ff53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:05.782033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.782015 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:05.782364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.782339 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq"] Oct 30 18:13:05.782606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.782565 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" containerID="cri-o://804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18" gracePeriod=30 Oct 30 18:13:05.783948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.783920 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-xks-ca" pod "c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" (UID: "c8f4c4cf-2eb2-4490-8719-795ea7f3ff53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:05.784060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.783942 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-kube-api-access-nscms" pod "c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" (UID: "c8f4c4cf-2eb2-4490-8719-795ea7f3ff53"). InnerVolumeSpecName "kube-api-access-nscms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:05.882833 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.882788 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscms\" (UniqueName: \"kubernetes.io/projected/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-kube-api-access-nscms\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:05.883000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:05.882847 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:06.223702 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:13:06.223668 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:13:06.325275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.325240 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh"] Oct 30 18:13:06.326151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.326125 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" containerName="revoke-keys" Oct 30 18:13:06.326151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.326152 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" containerName="revoke-keys" Oct 30 18:13:06.326791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.326300 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" containerName="revoke-keys" Oct 30 18:13:06.330395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.330373 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.346781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.346756 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh"] Oct 30 18:13:06.351860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.351806 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"openshift-service-ca.crt\"" Oct 30 18:13:06.351976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.351806 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"kube-root-ca.crt\"" Oct 30 18:13:06.351976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.351908 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"default-dockercfg-w2r27\"" Oct 30 18:13:06.356172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.356148 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"distinct-w1-380c08-kserve-self-signed-certs\"" Oct 30 18:13:06.369408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.369371 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:06.387178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.387157 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tmp-dir\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.387291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.387214 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-home\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.387356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.387288 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-dshm\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.387450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.387428 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-model-cache\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.387489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.387465 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tls-certs\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.387605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.387589 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ml4w\" (UniqueName: \"kubernetes.io/projected/3260d1a7-a604-4bc2-b904-a2fe7129bd23-kube-api-access-9ml4w\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.488324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.488232 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-home\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.488324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.488270 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-dshm\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.488576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.488344 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-model-cache\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.488576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.488371 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tls-certs\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.488576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.488434 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ml4w\" (UniqueName: \"kubernetes.io/projected/3260d1a7-a604-4bc2-b904-a2fe7129bd23-kube-api-access-9ml4w\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.488576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.488538 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tmp-dir\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.488791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.488690 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-home\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.489031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.489008 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-model-cache\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.489133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.489029 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tmp-dir\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.490896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.490869 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-dshm\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.491029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.491008 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tls-certs\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.509596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.509543 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ml4w\" (UniqueName: \"kubernetes.io/projected/3260d1a7-a604-4bc2-b904-a2fe7129bd23-kube-api-access-9ml4w\") pod \"distinct-w1-380c08-kserve-66ff8cc4-8f5bh\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.535732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.535354 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" event={"ID":"c8f4c4cf-2eb2-4490-8719-795ea7f3ff53","Type":"ContainerDied","Data":"f1724b0753e6483eb32e53d51c2b565cbe7adf6625f41eb3e6000c24368b6d66"} Oct 30 18:13:06.535732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.535503 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1724b0753e6483eb32e53d51c2b565cbe7adf6625f41eb3e6000c24368b6d66" Oct 30 18:13:06.535732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.535591 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx" Oct 30 18:13:06.647747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.647717 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:06.949544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:06.949512 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh"] Oct 30 18:13:06.952848 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:13:06.952789 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3260d1a7_a604_4bc2_b904_a2fe7129bd23.slice/crio-177dcaa631c7b3bbcc669aa32646beb3b8aa48b4a9938543c0785b8dbc3e389d WatchSource:0}: Error finding container 177dcaa631c7b3bbcc669aa32646beb3b8aa48b4a9938543c0785b8dbc3e389d: Status 404 returned error can't find the container with id 177dcaa631c7b3bbcc669aa32646beb3b8aa48b4a9938543c0785b8dbc3e389d Oct 30 18:13:07.350308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:07.350212 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:07.543740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:07.543703 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" event={"ID":"3260d1a7-a604-4bc2-b904-a2fe7129bd23","Type":"ContainerStarted","Data":"2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77"} Oct 30 18:13:07.543740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:07.543744 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" event={"ID":"3260d1a7-a604-4bc2-b904-a2fe7129bd23","Type":"ContainerStarted","Data":"177dcaa631c7b3bbcc669aa32646beb3b8aa48b4a9938543c0785b8dbc3e389d"} Oct 30 18:13:07.580775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:07.580725 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podStartSLOduration=1.5807120129999999 podStartE2EDuration="1.580712013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:07.57823417 +0000 UTC m=+884.402628487" watchObservedRunningTime="2026-10-30 18:13:07.580712013 +0000 UTC m=+884.405106304" Oct 30 18:13:07.648213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:07.648186 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:09.003151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:09.003117 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:10.648771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:10.648735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:11.062397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.062371 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:13:11.145151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145120 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-token\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-token\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145175 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-credential-socket\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145208 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-envoy\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145236 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-certs\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145302 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-data\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145377 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-podinfo\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145438 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-socket\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145450 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-credential-socket" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:11.145670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145498 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istiod-ca-cert\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145657 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-kube-api-access-kxfg5\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-kube-api-access-kxfg5\") pod \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\" (UID: \"1fc25d92-99d6-40db-9d7f-c2382e18d23b\") " Oct 30 18:13:11.145784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145642 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-certs" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:11.146068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145911 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-data" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:11.146068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.145991 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istiod-ca-cert" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:11.146068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.146016 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-socket" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:11.146313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.146210 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.146313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.146230 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.146313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.146246 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.146313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.146260 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.146313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.146274 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.147981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.147925 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-podinfo" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:13:11.148099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.147983 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-token" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:11.148099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.148025 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-envoy" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:11.148302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.148288 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-kube-api-access-kxfg5" pod "1fc25d92-99d6-40db-9d7f-c2382e18d23b" (UID: "1fc25d92-99d6-40db-9d7f-c2382e18d23b"). InnerVolumeSpecName "kube-api-access-kxfg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:11.247025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.246993 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxfg5\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-kube-api-access-kxfg5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.247025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.247021 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.247025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.247031 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.247238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.247038 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc25d92-99d6-40db-9d7f-c2382e18d23b-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.571351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.571272 4582 generic.go:356] "Generic (PLEG): container finished" podID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerID="804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18" exitCode=0 Oct 30 18:13:11.571351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.571318 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" Oct 30 18:13:11.571523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.571349 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" event={"ID":"1fc25d92-99d6-40db-9d7f-c2382e18d23b","Type":"ContainerDied","Data":"804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18"} Oct 30 18:13:11.571523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.571403 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq" event={"ID":"1fc25d92-99d6-40db-9d7f-c2382e18d23b","Type":"ContainerDied","Data":"479b4080e774ab240fba64a55fc1d7f3f148c17961701ed65957dadd8d813e9c"} Oct 30 18:13:11.571523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.571424 4582 scope.go:122] "RemoveContainer" containerID="804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18" Oct 30 18:13:11.583358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.583341 4582 scope.go:122] "RemoveContainer" containerID="804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18" Oct 30 18:13:11.583628 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:13:11.583607 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18\": container with ID starting with 804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18 not found: ID does not exist" containerID="804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18" Oct 30 18:13:11.583718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.583635 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18"} err="failed to get container status \"804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18\": rpc error: code = NotFound desc = could not find container \"804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18\": container with ID starting with 804169ada30af0ac772b983ff1f33ee9c7601ad4b8907a7b10a89ea1b3531b18 not found: ID does not exist" Oct 30 18:13:11.624251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.624223 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq"] Oct 30 18:13:11.633353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:11.633328 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xkvhq"] Oct 30 18:13:12.196594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:12.196564 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:13.026471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:13.026442 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:13.299209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:13.299134 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" path="/var/lib/kubelet/pods/1fc25d92-99d6-40db-9d7f-c2382e18d23b/volumes" Oct 30 18:13:13.670231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:13.670196 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:13.786975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:13.786906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:15.622224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:15.622192 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:16.648154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:16.648128 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:16.660763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:16.660736 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:17.128751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:17.128722 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:17.621141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:17.621107 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:13:17.675216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:17.675180 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:18.731066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:18.731033 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:20.246006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:20.245968 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:21.842475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:21.842442 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:23.493651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:23.493616 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:25.023276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:25.023233 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:25.322857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:25.322742 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:26.738069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:26.738038 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:27.833863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.833834 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk"] Oct 30 18:13:27.834403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.834387 4582 cpu_manager.go:414] "containerMap: removing container" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" Oct 30 18:13:27.834470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.834404 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" Oct 30 18:13:27.834527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.834492 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc25d92-99d6-40db-9d7f-c2382e18d23b" containerName="istio-proxy" Oct 30 18:13:27.837738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.837720 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:27.855770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.855742 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk"] Oct 30 18:13:27.859280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.859248 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"distinct-w4-fe1374-kserve-self-signed-certs\"" Oct 30 18:13:27.880058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.880009 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:27.937900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.937867 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-model-cache\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:27.938108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.938007 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-home\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:27.938108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.938038 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0c1119-b013-48be-8604-a21ed73bf11e-tls-certs\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:27.938210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.938194 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-tmp-dir\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:27.938248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.938221 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hpn8\" (UniqueName: \"kubernetes.io/projected/0d0c1119-b013-48be-8604-a21ed73bf11e-kube-api-access-5hpn8\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:27.938285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:27.938249 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-dshm\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.039959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.039929 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-tmp-dir\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.039959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.039964 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hpn8\" (UniqueName: \"kubernetes.io/projected/0d0c1119-b013-48be-8604-a21ed73bf11e-kube-api-access-5hpn8\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.040246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.039991 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-dshm\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.040246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.040044 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-model-cache\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.040246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.040149 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-home\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.040394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.040290 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0c1119-b013-48be-8604-a21ed73bf11e-tls-certs\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.040394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.040374 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-tmp-dir\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.040476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.040456 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-home\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.040532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.040487 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-model-cache\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.042274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.042250 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-dshm\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.042810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.042788 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0c1119-b013-48be-8604-a21ed73bf11e-tls-certs\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.057808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.057779 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hpn8\" (UniqueName: \"kubernetes.io/projected/0d0c1119-b013-48be-8604-a21ed73bf11e-kube-api-access-5hpn8\") pod \"distinct-w4-fe1374-kserve-56568bfdc-pwmwk\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.150134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.150106 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:28.448001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.447922 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk"] Oct 30 18:13:28.451207 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:13:28.451180 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d0c1119_b013_48be_8604_a21ed73bf11e.slice/crio-99a01754477d31ac06b55e702efcf29963fae96acd6911a79c39a559a98ebc14 WatchSource:0}: Error finding container 99a01754477d31ac06b55e702efcf29963fae96acd6911a79c39a559a98ebc14: Status 404 returned error can't find the container with id 99a01754477d31ac06b55e702efcf29963fae96acd6911a79c39a559a98ebc14 Oct 30 18:13:28.463381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.463353 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:28.677393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.677355 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" event={"ID":"0d0c1119-b013-48be-8604-a21ed73bf11e","Type":"ContainerStarted","Data":"d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c"} Oct 30 18:13:28.677560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.677401 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" event={"ID":"0d0c1119-b013-48be-8604-a21ed73bf11e","Type":"ContainerStarted","Data":"99a01754477d31ac06b55e702efcf29963fae96acd6911a79c39a559a98ebc14"} Oct 30 18:13:28.712144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:28.712055 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podStartSLOduration=1.712037756 podStartE2EDuration="1.712037756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:28.709804067 +0000 UTC m=+905.534198360" watchObservedRunningTime="2026-10-30 18:13:28.712037756 +0000 UTC m=+905.536432049" Oct 30 18:13:29.150339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:29.150310 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:30.040110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:30.040078 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:31.787682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:31.787636 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:31.949953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:31.944347 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:33.669109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:33.669076 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:35.232003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:35.231962 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:36.225522 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:13:36.225489 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:13:36.919655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:36.919608 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:38.151317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:38.151284 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:38.164424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:38.164401 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:38.431130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:38.431052 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:38.745727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:38.745646 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:13:40.000888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:40.000857 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:41.575444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:41.575405 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:43.356104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:43.356072 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:44.101552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:44.101520 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:44.940356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:44.940303 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:46.458298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:46.458265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:48.124754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:48.124722 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:49.121354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.121321 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d"] Oct 30 18:13:49.125907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.125880 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.142519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.142490 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d"] Oct 30 18:13:49.147810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.147791 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"distinct-2-w1-380c08-kserve-self-signed-certs\"" Oct 30 18:13:49.177530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.177496 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-model-cache\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.177651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.177533 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-home\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.177651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.177560 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-dshm\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.177757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.177745 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-tmp-dir\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.177793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.177783 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf04596b-f187-4b45-b528-2e6698a13711-tls-certs\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.177850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.177801 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l7s7\" (UniqueName: \"kubernetes.io/projected/cf04596b-f187-4b45-b528-2e6698a13711-kube-api-access-8l7s7\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.180467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.180440 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:49.279337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279311 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-tmp-dir\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279357 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf04596b-f187-4b45-b528-2e6698a13711-tls-certs\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279374 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7s7\" (UniqueName: \"kubernetes.io/projected/cf04596b-f187-4b45-b528-2e6698a13711-kube-api-access-8l7s7\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279410 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-model-cache\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279426 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-home\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279460 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-dshm\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279746 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-tmp-dir\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279814 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-model-cache\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.279869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.279853 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-home\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.281586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.281566 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-dshm\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.281831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.281803 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf04596b-f187-4b45-b528-2e6698a13711-tls-certs\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.295182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.295162 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7s7\" (UniqueName: \"kubernetes.io/projected/cf04596b-f187-4b45-b528-2e6698a13711-kube-api-access-8l7s7\") pod \"distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.442280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.442209 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:49.738975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.738885 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d"] Oct 30 18:13:49.740680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.740643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:49.742956 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:13:49.742924 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf04596b_f187_4b45_b528_2e6698a13711.slice/crio-7ac269dab4d6c4c5c6f3fc8febbb8c0851cfa674bc2834c7e9133343a57ab7d7 WatchSource:0}: Error finding container 7ac269dab4d6c4c5c6f3fc8febbb8c0851cfa674bc2834c7e9133343a57ab7d7: Status 404 returned error can't find the container with id 7ac269dab4d6c4c5c6f3fc8febbb8c0851cfa674bc2834c7e9133343a57ab7d7 Oct 30 18:13:49.807773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.807741 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" event={"ID":"cf04596b-f187-4b45-b528-2e6698a13711","Type":"ContainerStarted","Data":"2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a"} Oct 30 18:13:49.807773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.807783 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" event={"ID":"cf04596b-f187-4b45-b528-2e6698a13711","Type":"ContainerStarted","Data":"7ac269dab4d6c4c5c6f3fc8febbb8c0851cfa674bc2834c7e9133343a57ab7d7"} Oct 30 18:13:49.842578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:49.842518 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podStartSLOduration=0.842503428 podStartE2EDuration="842.503428ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:49.840423793 +0000 UTC m=+926.664818085" watchObservedRunningTime="2026-10-30 18:13:49.842503428 +0000 UTC m=+926.666897761" Oct 30 18:13:50.442920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:50.442885 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:51.558841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:51.558336 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:53.347660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:53.347626 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:54.834530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:54.834495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:55.016398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:55.016366 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:56.647742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:56.647707 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:58.225523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:58.225471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:59.442436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:59.442404 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:59.455075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:59.455048 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:13:59.732587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:59.732510 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:59.890441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:13:59.890406 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:14:01.335359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.335329 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:01.605813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.605737 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x"] Oct 30 18:14:01.609925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.609908 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.643263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.643232 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x"] Oct 30 18:14:01.742315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742267 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742333 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742470 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742516 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8csvz\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-kube-api-access-8csvz\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742539 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742672 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742781 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742811 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92b1954-580c-4d33-8fd9-e9e54ada9670-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.742935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.742893 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844024 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844063 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92b1954-580c-4d33-8fd9-e9e54ada9670-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844086 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844228 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844299 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844436 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844467 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844496 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8csvz\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-kube-api-access-8csvz\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844534 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844579 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844601 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844724 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844744 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.844932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.844909 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92b1954-580c-4d33-8fd9-e9e54ada9670-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.846675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.846652 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.846851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.846815 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.861136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.861071 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.861251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.861233 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8csvz\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-kube-api-access-8csvz\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:01.939740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:01.939716 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:02.250689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.250658 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x"] Oct 30 18:14:02.254151 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:02.254121 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92b1954_580c_4d33_8fd9_e9e54ada9670.slice/crio-c49a53e677cd50997b13653cceabfc737cfa2f2e351063737175d70071c7ff0c WatchSource:0}: Error finding container c49a53e677cd50997b13653cceabfc737cfa2f2e351063737175d70071c7ff0c: Status 404 returned error can't find the container with id c49a53e677cd50997b13653cceabfc737cfa2f2e351063737175d70071c7ff0c Oct 30 18:14:02.256548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.256519 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:02.256609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.256585 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:02.256644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.256616 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:02.901017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.900981 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" event={"ID":"d92b1954-580c-4d33-8fd9-e9e54ada9670","Type":"ContainerStarted","Data":"91f1852936cd5c4fb7f928265377fb71ee85449cd790fad7434c248f9ffe789e"} Oct 30 18:14:02.901413 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.901039 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" event={"ID":"d92b1954-580c-4d33-8fd9-e9e54ada9670","Type":"ContainerStarted","Data":"c49a53e677cd50997b13653cceabfc737cfa2f2e351063737175d70071c7ff0c"} Oct 30 18:14:02.933792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.933771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:02.940647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.940629 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:02.951647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:02.951591 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podStartSLOduration=1.9515718149999999 podStartE2EDuration="1.951571815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:02.945976714 +0000 UTC m=+939.770371004" watchObservedRunningTime="2026-10-30 18:14:02.951571815 +0000 UTC m=+939.775966108" Oct 30 18:14:03.950974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:03.950937 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:03.951595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:03.951556 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:04.742671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:04.742593 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:04.940311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:04.940280 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:04.940519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:04.940328 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:05.940974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:05.940933 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:05.941503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:05.940985 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:06.224938 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:14:06.224845 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:14:06.367440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:06.367384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:06.940887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:06.940852 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:06.941101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:06.940925 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:07.940345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:07.940314 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:07.940520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:07.940371 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:08.022268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:08.022236 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:09.172133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:09.172102 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:09.172554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:09.172489 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:09.173318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:09.173299 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:14:09.540509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:09.540431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:10.148297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:10.148265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:11.167941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.167890 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:11.710965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.710933 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl"] Oct 30 18:14:11.715512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.715489 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.743039 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.743008 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl"] Oct 30 18:14:11.776610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.776583 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.776771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.776651 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.776771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.776740 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.776922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.776812 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.776922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.776901 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.776997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.776976 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.777037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.777008 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.777091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.777040 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.777148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.777128 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8t7\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-kube-api-access-gh8t7\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878265 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878330 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878369 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878401 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8t7\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-kube-api-access-gh8t7\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878440 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878534 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878564 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878635 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878682 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878745 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.878945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878810 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.879066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.878971 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.879127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.879106 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.879195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.879169 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.880938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.880910 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.881326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.881309 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.895494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.895457 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.896339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.896315 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8t7\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-kube-api-access-gh8t7\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:11.962607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:11.962501 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:12.029262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.029228 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:12.341262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.341235 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl"] Oct 30 18:14:12.344207 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:12.344181 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ef62a5_21f7_4ef5_9980_1057fcebc3fa.slice/crio-bfb5998d3c4372c7b6adf2c83b57350cf6a034ca442566d4a1b746eb822a2da9 WatchSource:0}: Error finding container bfb5998d3c4372c7b6adf2c83b57350cf6a034ca442566d4a1b746eb822a2da9: Status 404 returned error can't find the container with id bfb5998d3c4372c7b6adf2c83b57350cf6a034ca442566d4a1b746eb822a2da9 Oct 30 18:14:12.347145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.347087 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:12.347319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.347292 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:12.347404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.347387 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:12.753957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.753912 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:12.972275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.972242 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" event={"ID":"87ef62a5-21f7-4ef5-9980-1057fcebc3fa","Type":"ContainerStarted","Data":"2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6"} Oct 30 18:14:12.972562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:12.972522 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" event={"ID":"87ef62a5-21f7-4ef5-9980-1057fcebc3fa","Type":"ContainerStarted","Data":"bfb5998d3c4372c7b6adf2c83b57350cf6a034ca442566d4a1b746eb822a2da9"} Oct 30 18:14:13.020251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:13.020148 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podStartSLOduration=2.02013332 podStartE2EDuration="2.02013332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:13.016751326 +0000 UTC m=+949.841145620" watchObservedRunningTime="2026-10-30 18:14:13.02013332 +0000 UTC m=+949.844527612" Oct 30 18:14:13.030347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:13.030315 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:13.031596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:13.031574 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:13.031697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:13.031616 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:13.128287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:13.128254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:14.029680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:14.029650 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:14.030183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:14.029706 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:14.501006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:14.500970 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:15.029668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:15.029637 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:15.029847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:15.029692 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:16.029691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.029661 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:16.029890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.029710 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:16.033906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.033872 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:16.920087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.920046 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk"] Oct 30 18:14:16.926246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.926217 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:16.939324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.939302 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk"] Oct 30 18:14:16.962055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.962029 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:14:16.965048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.963659 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:16.965225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.963731 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:14:16.965330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:16.963766 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:14:17.030397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.030362 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:17.030787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.030449 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:17.036108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.036083 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.036356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.036342 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65xq\" (UniqueName: \"kubernetes.io/projected/67a9221d-6334-4fd5-ba87-0c50a64e78fe-kube-api-access-w65xq\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.036475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.036461 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.137916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.137874 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.138122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.138061 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.138172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.138153 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65xq\" (UniqueName: \"kubernetes.io/projected/67a9221d-6334-4fd5-ba87-0c50a64e78fe-kube-api-access-w65xq\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.138709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.138686 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.140499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.140479 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.163841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.162161 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65xq\" (UniqueName: \"kubernetes.io/projected/67a9221d-6334-4fd5-ba87-0c50a64e78fe-kube-api-access-w65xq\") pod \"maas-api-revoke-sub-6a5da84a-tfzsk\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.262350 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.262020 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:17.572954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.572648 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk"] Oct 30 18:14:17.579709 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:17.579243 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a9221d_6334_4fd5_ba87_0c50a64e78fe.slice/crio-3f280a8f07cada97fb594b2d3f18627b77be0f4b021560330771230d24cf75a8 WatchSource:0}: Error finding container 3f280a8f07cada97fb594b2d3f18627b77be0f4b021560330771230d24cf75a8: Status 404 returned error can't find the container with id 3f280a8f07cada97fb594b2d3f18627b77be0f4b021560330771230d24cf75a8 Oct 30 18:14:17.664120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:17.663993 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:18.006137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:18.006099 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" event={"ID":"67a9221d-6334-4fd5-ba87-0c50a64e78fe","Type":"ContainerStarted","Data":"bd8c12d196e8034d7eae9316ea541ab6966ef00a9b815677439aa5e9ea073b12"} Oct 30 18:14:18.006296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:18.006160 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" event={"ID":"67a9221d-6334-4fd5-ba87-0c50a64e78fe","Type":"ContainerStarted","Data":"3f280a8f07cada97fb594b2d3f18627b77be0f4b021560330771230d24cf75a8"} Oct 30 18:14:18.030461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:18.030432 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:18.030795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:18.030476 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:18.039588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:18.039546 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" podStartSLOduration=2.039534729 podStartE2EDuration="2.039534729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:18.038901108 +0000 UTC m=+954.863295420" watchObservedRunningTime="2026-10-30 18:14:18.039534729 +0000 UTC m=+954.863929071" Oct 30 18:14:19.014350 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:19.014313 4582 generic.go:356] "Generic (PLEG): container finished" podID="67a9221d-6334-4fd5-ba87-0c50a64e78fe" containerID="bd8c12d196e8034d7eae9316ea541ab6966ef00a9b815677439aa5e9ea073b12" exitCode=0 Oct 30 18:14:19.014539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:19.014388 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" event={"ID":"67a9221d-6334-4fd5-ba87-0c50a64e78fe","Type":"ContainerDied","Data":"bd8c12d196e8034d7eae9316ea541ab6966ef00a9b815677439aa5e9ea073b12"} Oct 30 18:14:19.029454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:19.029432 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:19.029577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:19.029473 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:19.336255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:19.336123 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:20.035473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.035445 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:20.035885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.035862 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:20.036550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.036529 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:14:20.181515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.181488 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:20.290671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.290600 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-xks-ca\") pod \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " Oct 30 18:14:20.290845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.290674 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a9221d-6334-4fd5-ba87-0c50a64e78fe-kube-api-access-w65xq\" (UniqueName: \"kubernetes.io/projected/67a9221d-6334-4fd5-ba87-0c50a64e78fe-kube-api-access-w65xq\") pod \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " Oct 30 18:14:20.290889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.290843 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-openshift-ca\") pod \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\" (UID: \"67a9221d-6334-4fd5-ba87-0c50a64e78fe\") " Oct 30 18:14:20.291179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.291147 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-openshift-ca" pod "67a9221d-6334-4fd5-ba87-0c50a64e78fe" (UID: "67a9221d-6334-4fd5-ba87-0c50a64e78fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:20.291393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.291376 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:20.292743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.292715 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-xks-ca" pod "67a9221d-6334-4fd5-ba87-0c50a64e78fe" (UID: "67a9221d-6334-4fd5-ba87-0c50a64e78fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:20.292902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.292763 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a9221d-6334-4fd5-ba87-0c50a64e78fe-kube-api-access-w65xq" pod "67a9221d-6334-4fd5-ba87-0c50a64e78fe" (UID: "67a9221d-6334-4fd5-ba87-0c50a64e78fe"). InnerVolumeSpecName "kube-api-access-w65xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:20.393465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.393438 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a9221d-6334-4fd5-ba87-0c50a64e78fe-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:20.393465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.393463 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w65xq\" (UniqueName: \"kubernetes.io/projected/67a9221d-6334-4fd5-ba87-0c50a64e78fe-kube-api-access-w65xq\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:20.841652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:20.841622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:21.028256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:21.028215 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" event={"ID":"67a9221d-6334-4fd5-ba87-0c50a64e78fe","Type":"ContainerDied","Data":"3f280a8f07cada97fb594b2d3f18627b77be0f4b021560330771230d24cf75a8"} Oct 30 18:14:21.028256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:21.028255 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk" Oct 30 18:14:21.028467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:21.028297 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f280a8f07cada97fb594b2d3f18627b77be0f4b021560330771230d24cf75a8" Oct 30 18:14:22.322091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:22.322057 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:23.665797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.665762 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w"] Oct 30 18:14:23.666572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.666543 4582 cpu_manager.go:414] "containerMap: removing container" podUID="67a9221d-6334-4fd5-ba87-0c50a64e78fe" containerName="revoke-keys" Oct 30 18:14:23.666572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.666569 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a9221d-6334-4fd5-ba87-0c50a64e78fe" containerName="revoke-keys" Oct 30 18:14:23.666768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.666662 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a9221d-6334-4fd5-ba87-0c50a64e78fe" containerName="revoke-keys" Oct 30 18:14:23.672338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.671700 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.699866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.698126 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w"] Oct 30 18:14:23.701717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.701692 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-921450-openshift-default-dockercfg-rt5dw\"" Oct 30 18:14:23.834506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834474 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.834506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834511 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.834799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834536 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.834799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834637 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.834799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834669 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.834799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834783 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.835054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834865 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bed21e-b5ec-4411-b09c-456575cd00fc-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.835054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834921 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.835054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.834968 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7fg\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-kube-api-access-rb7fg\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936141 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bed21e-b5ec-4411-b09c-456575cd00fc-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936192 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936232 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7fg\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-kube-api-access-rb7fg\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936268 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936297 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936333 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936367 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936396 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936489 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936762 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936812 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.936885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.936876 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.937131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.937104 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bed21e-b5ec-4411-b09c-456575cd00fc-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.937224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.937201 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.939272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.939251 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.939616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.939595 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:23.968637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.968606 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:24.000090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:23.987531 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7fg\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-kube-api-access-rb7fg\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:24.021961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:24.021919 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:24.070273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:24.070237 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:24.419210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:24.415663 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w"] Oct 30 18:14:24.437451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:24.437411 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:24.438111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:24.437641 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:24.438111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:24.437696 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:25.097048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:25.097006 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" event={"ID":"63bed21e-b5ec-4411-b09c-456575cd00fc","Type":"ContainerStarted","Data":"f264d1f045b41f5607d08017ce6cc7a73da01e65589855eb642b4702a68f0171"} Oct 30 18:14:25.097631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:25.097077 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" event={"ID":"63bed21e-b5ec-4411-b09c-456575cd00fc","Type":"ContainerStarted","Data":"17f4df9b04402e36940f681ece83d70c1ec5fad9918502ff6f8def38a189983b"} Oct 30 18:14:26.023268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:26.023236 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:26.024712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:26.024683 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.024862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:26.024735 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:14:26.264000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:26.263964 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:26.537761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:26.537721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:27.022652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:27.022615 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.022841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:27.022671 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:14:27.994740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:27.994711 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:28.022878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:28.022850 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.023027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:28.022903 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:14:29.023082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:29.023044 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.023468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:29.023112 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:14:29.879159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:29.879125 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:30.023544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.023507 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.023992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.023558 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:14:30.718892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.718839 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podStartSLOduration=7.718811192 podStartE2EDuration="7.718811192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:25.18693294 +0000 UTC m=+962.011327253" watchObservedRunningTime="2026-10-30 18:14:30.718811192 +0000 UTC m=+967.543205484" Oct 30 18:14:30.719751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.719735 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt"] Oct 30 18:14:30.723938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.723921 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.737596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.737575 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:14:30.763846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.749992 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:30.763846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.750120 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:14:30.763846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.750706 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:14:30.779137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.779105 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt"] Oct 30 18:14:30.807884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.807855 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.808063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.807937 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfsp\" (UniqueName: \"kubernetes.io/projected/55c01f61-0115-452d-b93d-f8810966f95d-kube-api-access-cmfsp\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.808063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.808003 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.909675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.909639 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.909858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.909732 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfsp\" (UniqueName: \"kubernetes.io/projected/55c01f61-0115-452d-b93d-f8810966f95d-kube-api-access-cmfsp\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.909858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.909752 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.910378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.910356 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.912138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.912115 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:30.926169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:30.926146 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfsp\" (UniqueName: \"kubernetes.io/projected/55c01f61-0115-452d-b93d-f8810966f95d-kube-api-access-cmfsp\") pod \"maas-api-revoke-sub-ddc061a6-4twpt\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:31.023412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:31.023340 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:31.023559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:31.023426 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:14:31.037556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:31.037533 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:31.359491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:31.359466 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt"] Oct 30 18:14:31.362150 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:31.362121 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c01f61_0115_452d_b93d_f8810966f95d.slice/crio-86c64017b6cf8c4c26693c9f3fb4ce3f36780c2dc957724a1feabdeb84e9cb50 WatchSource:0}: Error finding container 86c64017b6cf8c4c26693c9f3fb4ce3f36780c2dc957724a1feabdeb84e9cb50: Status 404 returned error can't find the container with id 86c64017b6cf8c4c26693c9f3fb4ce3f36780c2dc957724a1feabdeb84e9cb50 Oct 30 18:14:31.451714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:31.451680 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:32.145290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:32.145249 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" event={"ID":"55c01f61-0115-452d-b93d-f8810966f95d","Type":"ContainerStarted","Data":"9c13cfc9e9dbea6ae70fe162a4af5b494fbd5b69338b60dfa38baf6b165f5354"} Oct 30 18:14:32.145800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:32.145317 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" event={"ID":"55c01f61-0115-452d-b93d-f8810966f95d","Type":"ContainerStarted","Data":"86c64017b6cf8c4c26693c9f3fb4ce3f36780c2dc957724a1feabdeb84e9cb50"} Oct 30 18:14:32.179506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:32.179336 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" podStartSLOduration=2.17931525 podStartE2EDuration="2.17931525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:32.17777522 +0000 UTC m=+969.002169513" watchObservedRunningTime="2026-10-30 18:14:32.17931525 +0000 UTC m=+969.003709544" Oct 30 18:14:32.184426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:32.184403 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:32.184800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:32.184780 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:32.185789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:32.185761 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:14:33.039635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:33.039574 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:33.161897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:33.161858 4582 generic.go:356] "Generic (PLEG): container finished" podID="55c01f61-0115-452d-b93d-f8810966f95d" containerID="9c13cfc9e9dbea6ae70fe162a4af5b494fbd5b69338b60dfa38baf6b165f5354" exitCode=0 Oct 30 18:14:33.162298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:33.161943 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" event={"ID":"55c01f61-0115-452d-b93d-f8810966f95d","Type":"ContainerDied","Data":"9c13cfc9e9dbea6ae70fe162a4af5b494fbd5b69338b60dfa38baf6b165f5354"} Oct 30 18:14:34.307223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.307199 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:34.350905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.350869 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-xks-ca\") pod \"55c01f61-0115-452d-b93d-f8810966f95d\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " Oct 30 18:14:34.351031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.350928 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-openshift-ca\") pod \"55c01f61-0115-452d-b93d-f8810966f95d\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " Oct 30 18:14:34.351183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.351164 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c01f61-0115-452d-b93d-f8810966f95d-kube-api-access-cmfsp\" (UniqueName: \"kubernetes.io/projected/55c01f61-0115-452d-b93d-f8810966f95d-kube-api-access-cmfsp\") pod \"55c01f61-0115-452d-b93d-f8810966f95d\" (UID: \"55c01f61-0115-452d-b93d-f8810966f95d\") " Oct 30 18:14:34.351568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.351540 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-openshift-ca" pod "55c01f61-0115-452d-b93d-f8810966f95d" (UID: "55c01f61-0115-452d-b93d-f8810966f95d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:34.351704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.351688 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:34.353181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.353150 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-xks-ca" pod "55c01f61-0115-452d-b93d-f8810966f95d" (UID: "55c01f61-0115-452d-b93d-f8810966f95d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:34.353600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.353577 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c01f61-0115-452d-b93d-f8810966f95d-kube-api-access-cmfsp" pod "55c01f61-0115-452d-b93d-f8810966f95d" (UID: "55c01f61-0115-452d-b93d-f8810966f95d"). InnerVolumeSpecName "kube-api-access-cmfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:34.452715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.452645 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfsp\" (UniqueName: \"kubernetes.io/projected/55c01f61-0115-452d-b93d-f8810966f95d-kube-api-access-cmfsp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:34.452715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.452673 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c01f61-0115-452d-b93d-f8810966f95d-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:34.621352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:34.621316 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:35.175414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:35.175390 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" Oct 30 18:14:35.175582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:35.175389 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt" event={"ID":"55c01f61-0115-452d-b93d-f8810966f95d","Type":"ContainerDied","Data":"86c64017b6cf8c4c26693c9f3fb4ce3f36780c2dc957724a1feabdeb84e9cb50"} Oct 30 18:14:35.175582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:35.175510 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c64017b6cf8c4c26693c9f3fb4ce3f36780c2dc957724a1feabdeb84e9cb50" Oct 30 18:14:35.278185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:35.278152 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:35.379415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:35.379378 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:36.224648 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:14:36.224605 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:14:36.273119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:36.273090 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:37.738844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.737593 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft"] Oct 30 18:14:37.742850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.742116 4582 cpu_manager.go:414] "containerMap: removing container" podUID="55c01f61-0115-452d-b93d-f8810966f95d" containerName="revoke-keys" Oct 30 18:14:37.742850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.742142 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c01f61-0115-452d-b93d-f8810966f95d" containerName="revoke-keys" Oct 30 18:14:37.742850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.742347 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c01f61-0115-452d-b93d-f8810966f95d" containerName="revoke-keys" Oct 30 18:14:37.777036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.777006 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft"] Oct 30 18:14:37.777390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.777316 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967339 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967393 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967453 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967495 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtqg\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-kube-api-access-fgtqg\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967532 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967588 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967632 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967708 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:37.982428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:37.967764 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.077062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.076933 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.077362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.077340 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.077549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.077531 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.086012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.085985 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.090846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.086199 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtqg\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-kube-api-access-fgtqg\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.091073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.091031 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.105009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.104981 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.105290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.105267 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.109846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.105508 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.114588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.080278 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.116911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.115062 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.117142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.116082 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.117274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.116318 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.117395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.116836 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.124380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.124352 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.134408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.134307 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.136885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.136806 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.137537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.137484 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtqg\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-kube-api-access-fgtqg\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.304248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.304203 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:38.625022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.624997 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft"] Oct 30 18:14:38.626728 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:38.626701 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb6f406_5b39_4e3a_ac81_1ac35f4dc583.slice/crio-9b3d1f691233dfc9169884c2b219ab6e269351791e5a500bdc18537e233d7da0 WatchSource:0}: Error finding container 9b3d1f691233dfc9169884c2b219ab6e269351791e5a500bdc18537e233d7da0: Status 404 returned error can't find the container with id 9b3d1f691233dfc9169884c2b219ab6e269351791e5a500bdc18537e233d7da0 Oct 30 18:14:38.629064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.629034 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:38.629148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.629097 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:38.629187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:38.629147 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:39.033539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.033460 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:39.232164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.232132 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" event={"ID":"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583","Type":"ContainerStarted","Data":"504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9"} Oct 30 18:14:39.232334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.232192 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" event={"ID":"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583","Type":"ContainerStarted","Data":"9b3d1f691233dfc9169884c2b219ab6e269351791e5a500bdc18537e233d7da0"} Oct 30 18:14:39.277949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.277715 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podStartSLOduration=2.277699477 podStartE2EDuration="2.277699477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:39.276799044 +0000 UTC m=+976.101193336" watchObservedRunningTime="2026-10-30 18:14:39.277699477 +0000 UTC m=+976.102093771" Oct 30 18:14:39.305285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.305218 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:39.306794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.306771 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:39.306937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.306814 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:14:39.764844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:39.764233 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:40.304626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:40.304593 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:40.305088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:40.304651 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:14:40.699361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:40.699322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:41.131317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.131284 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw"] Oct 30 18:14:41.135462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.135442 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.162106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.161062 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:41.162106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.161180 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:14:41.162106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.161239 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:14:41.162106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.161332 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:14:41.171449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.169500 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw"] Oct 30 18:14:41.275950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.275907 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66vr\" (UniqueName: \"kubernetes.io/projected/bfd67b9c-9b01-49fd-9958-8c1b165a6643-kube-api-access-v66vr\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.276127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.275975 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.276127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.276086 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.305001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.304974 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:41.305379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.305023 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:14:41.377634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.377589 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66vr\" (UniqueName: \"kubernetes.io/projected/bfd67b9c-9b01-49fd-9958-8c1b165a6643-kube-api-access-v66vr\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.377961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.377691 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.377961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.377790 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.378769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.378742 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.380137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.380115 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.396768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.396711 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66vr\" (UniqueName: \"kubernetes.io/projected/bfd67b9c-9b01-49fd-9958-8c1b165a6643-kube-api-access-v66vr\") pod \"maas-api-revoke-sub-fb289cc0-9hsnw\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.474814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.474780 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:41.775102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.775080 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw"] Oct 30 18:14:41.777419 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:41.777392 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd67b9c_9b01_49fd_9958_8c1b165a6643.slice/crio-7606c4b3b13944f1fdbb2b38c1d498d4fd7281f7b000b4b391df3edc8e9d8cc6 WatchSource:0}: Error finding container 7606c4b3b13944f1fdbb2b38c1d498d4fd7281f7b000b4b391df3edc8e9d8cc6: Status 404 returned error can't find the container with id 7606c4b3b13944f1fdbb2b38c1d498d4fd7281f7b000b4b391df3edc8e9d8cc6 Oct 30 18:14:41.872054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.872017 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b"] Oct 30 18:14:41.877989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.877968 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.885122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.885097 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b"] Oct 30 18:14:41.986013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.985989 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986045 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986108 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986171 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986222 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986257 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986285 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986412 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnw9\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-kube-api-access-gqnw9\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:41.986515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:41.986500 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087342 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087393 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087440 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087469 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087568 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnw9\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-kube-api-access-gqnw9\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087651 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087691 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087742 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087777 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.087902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.087887 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.088118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.088096 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.088231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.088203 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.088270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.088256 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.088485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.088455 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.089812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.089781 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.090100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.090082 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.103957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.103938 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.104100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.104083 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnw9\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-kube-api-access-gqnw9\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.197916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.197888 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:42.282615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.282576 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" event={"ID":"bfd67b9c-9b01-49fd-9958-8c1b165a6643","Type":"ContainerStarted","Data":"2c5223476734d5940d319b0f4675957aaa8d9542471ea37dae42a5f47dfee356"} Oct 30 18:14:42.282773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.282641 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" event={"ID":"bfd67b9c-9b01-49fd-9958-8c1b165a6643","Type":"ContainerStarted","Data":"7606c4b3b13944f1fdbb2b38c1d498d4fd7281f7b000b4b391df3edc8e9d8cc6"} Oct 30 18:14:42.306237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.305214 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:42.306237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.305263 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:14:42.323116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.323062 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" podStartSLOduration=1.323046636 podStartE2EDuration="1.323046636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:42.317540744 +0000 UTC m=+979.141935058" watchObservedRunningTime="2026-10-30 18:14:42.323046636 +0000 UTC m=+979.147440929" Oct 30 18:14:42.428565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.428534 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:42.533289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.533249 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b"] Oct 30 18:14:42.539237 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:42.539191 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4510a8c_2c41_45b4_a0fd_8f025c96c857.slice/crio-a55962321f6e5bf893f5a4f1448dad223c3fde8b14e7826679ea59dcd533b5b0 WatchSource:0}: Error finding container a55962321f6e5bf893f5a4f1448dad223c3fde8b14e7826679ea59dcd533b5b0: Status 404 returned error can't find the container with id a55962321f6e5bf893f5a4f1448dad223c3fde8b14e7826679ea59dcd533b5b0 Oct 30 18:14:42.543028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.542222 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:42.546838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.543608 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:42.546838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:42.543697 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:43.301306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:43.301230 4582 generic.go:356] "Generic (PLEG): container finished" podID="bfd67b9c-9b01-49fd-9958-8c1b165a6643" containerID="2c5223476734d5940d319b0f4675957aaa8d9542471ea37dae42a5f47dfee356" exitCode=0 Oct 30 18:14:43.301482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:43.301364 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" event={"ID":"bfd67b9c-9b01-49fd-9958-8c1b165a6643","Type":"ContainerDied","Data":"2c5223476734d5940d319b0f4675957aaa8d9542471ea37dae42a5f47dfee356"} Oct 30 18:14:43.302899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:43.302878 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" event={"ID":"f4510a8c-2c41-45b4-a0fd-8f025c96c857","Type":"ContainerStarted","Data":"f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4"} Oct 30 18:14:43.303006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:43.302912 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" event={"ID":"f4510a8c-2c41-45b4-a0fd-8f025c96c857","Type":"ContainerStarted","Data":"a55962321f6e5bf893f5a4f1448dad223c3fde8b14e7826679ea59dcd533b5b0"} Oct 30 18:14:43.305455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:43.305433 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.305513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:43.305477 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:14:44.198141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.198096 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:44.199511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.199487 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.199635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.199546 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:14:44.280843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.280729 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:44.305588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.305556 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.305745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.305608 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:14:44.604279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.604247 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:44.639083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.639038 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podStartSLOduration=3.639023147 podStartE2EDuration="3.639023147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:43.396812354 +0000 UTC m=+980.221206642" watchObservedRunningTime="2026-10-30 18:14:44.639023147 +0000 UTC m=+981.463417440" Oct 30 18:14:44.721580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.721548 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-xks-ca\") pod \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " Oct 30 18:14:44.721989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.721971 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd67b9c-9b01-49fd-9958-8c1b165a6643-kube-api-access-v66vr\" (UniqueName: \"kubernetes.io/projected/bfd67b9c-9b01-49fd-9958-8c1b165a6643-kube-api-access-v66vr\") pod \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " Oct 30 18:14:44.722182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.722170 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-openshift-ca\") pod \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\" (UID: \"bfd67b9c-9b01-49fd-9958-8c1b165a6643\") " Oct 30 18:14:44.722660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.722561 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-openshift-ca" pod "bfd67b9c-9b01-49fd-9958-8c1b165a6643" (UID: "bfd67b9c-9b01-49fd-9958-8c1b165a6643"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:44.722797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.722779 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:44.726146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.726126 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-xks-ca" pod "bfd67b9c-9b01-49fd-9958-8c1b165a6643" (UID: "bfd67b9c-9b01-49fd-9958-8c1b165a6643"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:44.726219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.726197 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd67b9c-9b01-49fd-9958-8c1b165a6643-kube-api-access-v66vr" pod "bfd67b9c-9b01-49fd-9958-8c1b165a6643" (UID: "bfd67b9c-9b01-49fd-9958-8c1b165a6643"). InnerVolumeSpecName "kube-api-access-v66vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:44.824208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.824126 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66vr\" (UniqueName: \"kubernetes.io/projected/bfd67b9c-9b01-49fd-9958-8c1b165a6643-kube-api-access-v66vr\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:44.824208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:44.824158 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd67b9c-9b01-49fd-9958-8c1b165a6643-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:45.199029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.198991 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.199455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.199049 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:14:45.304667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.304632 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.304856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.304682 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:14:45.316792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.316767 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" Oct 30 18:14:45.316924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.316796 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw" event={"ID":"bfd67b9c-9b01-49fd-9958-8c1b165a6643","Type":"ContainerDied","Data":"7606c4b3b13944f1fdbb2b38c1d498d4fd7281f7b000b4b391df3edc8e9d8cc6"} Oct 30 18:14:45.316924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.316864 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7606c4b3b13944f1fdbb2b38c1d498d4fd7281f7b000b4b391df3edc8e9d8cc6" Oct 30 18:14:45.985468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:45.985430 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:46.199003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.198974 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:46.199204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.199031 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:14:46.467984 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.459633 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:46.467984 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.467873 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:46.469489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.469464 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:14:46.611122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.611092 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz"] Oct 30 18:14:46.612032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.612011 4582 cpu_manager.go:414] "containerMap: removing container" podUID="bfd67b9c-9b01-49fd-9958-8c1b165a6643" containerName="revoke-keys" Oct 30 18:14:46.612091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.612036 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd67b9c-9b01-49fd-9958-8c1b165a6643" containerName="revoke-keys" Oct 30 18:14:46.612176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.612164 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd67b9c-9b01-49fd-9958-8c1b165a6643" containerName="revoke-keys" Oct 30 18:14:46.616252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.616231 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.628954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.628929 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:14:46.629116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.628987 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:46.629116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.629022 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:14:46.629243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.629151 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:14:46.633261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.633238 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz"] Oct 30 18:14:46.773990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.773606 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458q7\" (UniqueName: \"kubernetes.io/projected/34388c06-6a2e-4b53-b2e9-548a99e2535f-kube-api-access-458q7\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.773990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.773656 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.773990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.773725 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.781740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.781698 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr"] Oct 30 18:14:46.786998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.786974 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.791755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.791690 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr"] Oct 30 18:14:46.874440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.874406 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.874613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.874584 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458q7\" (UniqueName: \"kubernetes.io/projected/34388c06-6a2e-4b53-b2e9-548a99e2535f-kube-api-access-458q7\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.874613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.874608 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.875108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.875079 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.876991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.876960 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.900302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.900261 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458q7\" (UniqueName: \"kubernetes.io/projected/34388c06-6a2e-4b53-b2e9-548a99e2535f-kube-api-access-458q7\") pod \"maas-api-revoke-sub-9af32be1-ws6bz\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.931560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.931527 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:46.976137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976096 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976208 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976276 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976315 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976433 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976460 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976526 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976555 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:46.976734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:46.976605 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm68k\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-kube-api-access-jm68k\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.077941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.077839 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.077941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.077888 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.077941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.077909 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.077941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.077938 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.078343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.077991 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.078343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.078008 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.078343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.078030 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.078343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.078047 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.078343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.078085 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm68k\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-kube-api-access-jm68k\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.078599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.078519 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.082683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.080423 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.082683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.080692 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.082683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.080948 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.082683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.081190 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.082683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.081409 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.082683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.081480 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.096051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.096021 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.097770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.097752 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm68k\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-kube-api-access-jm68k\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.107441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.107419 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:47.199902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.198886 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:47.199902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.198948 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:14:47.276116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.274813 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz"] Oct 30 18:14:47.278494 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:47.278081 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34388c06_6a2e_4b53_b2e9_548a99e2535f.slice/crio-24144646b954dba711e3e39e905ffd923e194e498399b2f173db067e795af714 WatchSource:0}: Error finding container 24144646b954dba711e3e39e905ffd923e194e498399b2f173db067e795af714: Status 404 returned error can't find the container with id 24144646b954dba711e3e39e905ffd923e194e498399b2f173db067e795af714 Oct 30 18:14:47.331265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.331203 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" event={"ID":"34388c06-6a2e-4b53-b2e9-548a99e2535f","Type":"ContainerStarted","Data":"24144646b954dba711e3e39e905ffd923e194e498399b2f173db067e795af714"} Oct 30 18:14:47.439926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.439896 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr"] Oct 30 18:14:47.469163 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.469125 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:47.469386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.469362 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:47.469525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.469508 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:14:47.861845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:47.858735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:48.199099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:48.198610 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:48.199099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:48.198673 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:14:48.340066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:48.340028 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" event={"ID":"34388c06-6a2e-4b53-b2e9-548a99e2535f","Type":"ContainerStarted","Data":"93e4ed4e671013fece0c50c433d2a40aa3044a304eeb220693da5f830f689959"} Oct 30 18:14:48.363845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:48.341857 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" event={"ID":"fa4187a2-d01f-487d-8ebe-5c835efc4eee","Type":"ContainerStarted","Data":"b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0"} Oct 30 18:14:48.363845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:48.341907 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" event={"ID":"fa4187a2-d01f-487d-8ebe-5c835efc4eee","Type":"ContainerStarted","Data":"74710e55d359fbed988c46c909ab3a101eaa4e703d2f054105ed9ec2f2956aa6"} Oct 30 18:14:48.396996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:48.396938 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" podStartSLOduration=2.396921484 podStartE2EDuration="2.396921484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:48.39033796 +0000 UTC m=+985.214732284" watchObservedRunningTime="2026-10-30 18:14:48.396921484 +0000 UTC m=+985.221315776" Oct 30 18:14:48.429624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:48.429568 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podStartSLOduration=2.429556047 podStartE2EDuration="2.429556047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:48.426435782 +0000 UTC m=+985.250830070" watchObservedRunningTime="2026-10-30 18:14:48.429556047 +0000 UTC m=+985.253950339" Oct 30 18:14:49.108581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.108541 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:49.110475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.110446 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:49.110592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.110497 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:14:49.198540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.198505 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:49.198685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.198560 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:14:49.372893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.372850 4582 generic.go:356] "Generic (PLEG): container finished" podID="34388c06-6a2e-4b53-b2e9-548a99e2535f" containerID="93e4ed4e671013fece0c50c433d2a40aa3044a304eeb220693da5f830f689959" exitCode=0 Oct 30 18:14:49.373375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.372911 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" event={"ID":"34388c06-6a2e-4b53-b2e9-548a99e2535f","Type":"ContainerDied","Data":"93e4ed4e671013fece0c50c433d2a40aa3044a304eeb220693da5f830f689959"} Oct 30 18:14:49.628563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:49.628471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:50.021903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.021816 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq"] Oct 30 18:14:50.026025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.026003 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.106784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.105372 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq"] Oct 30 18:14:50.110378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.110340 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:50.110514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.110403 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:14:50.119274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.119246 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7bx\" (UniqueName: \"kubernetes.io/projected/d36f3041-677f-4970-b8bb-a3f5bc19f167-kube-api-access-qq7bx\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.119398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.119281 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.119475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.119396 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.202785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.202753 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:50.203189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.203163 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:50.203802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.203782 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:14:50.220901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.220873 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7bx\" (UniqueName: \"kubernetes.io/projected/d36f3041-677f-4970-b8bb-a3f5bc19f167-kube-api-access-qq7bx\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.221010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.220916 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.221182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.221162 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.221807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.221759 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.223260 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.223240 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.240655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.240628 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7bx\" (UniqueName: \"kubernetes.io/projected/d36f3041-677f-4970-b8bb-a3f5bc19f167-kube-api-access-qq7bx\") pod \"maas-api-revoke-sub-329b5113-9bdhq\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.341892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.341794 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:50.606215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.606147 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:50.626937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.626899 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34388c06-6a2e-4b53-b2e9-548a99e2535f-kube-api-access-458q7\" (UniqueName: \"kubernetes.io/projected/34388c06-6a2e-4b53-b2e9-548a99e2535f-kube-api-access-458q7\") pod \"34388c06-6a2e-4b53-b2e9-548a99e2535f\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " Oct 30 18:14:50.627059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.627020 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-openshift-ca\") pod \"34388c06-6a2e-4b53-b2e9-548a99e2535f\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " Oct 30 18:14:50.627117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.627101 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-xks-ca\") pod \"34388c06-6a2e-4b53-b2e9-548a99e2535f\" (UID: \"34388c06-6a2e-4b53-b2e9-548a99e2535f\") " Oct 30 18:14:50.627491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.627444 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-openshift-ca" pod "34388c06-6a2e-4b53-b2e9-548a99e2535f" (UID: "34388c06-6a2e-4b53-b2e9-548a99e2535f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:50.627676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.627656 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.629235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.629208 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34388c06-6a2e-4b53-b2e9-548a99e2535f-kube-api-access-458q7" pod "34388c06-6a2e-4b53-b2e9-548a99e2535f" (UID: "34388c06-6a2e-4b53-b2e9-548a99e2535f"). InnerVolumeSpecName "kube-api-access-458q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:50.629340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.629290 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-xks-ca" pod "34388c06-6a2e-4b53-b2e9-548a99e2535f" (UID: "34388c06-6a2e-4b53-b2e9-548a99e2535f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:50.704005 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.703983 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq"] Oct 30 18:14:50.706018 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:14:50.705991 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36f3041_677f_4970_b8bb_a3f5bc19f167.slice/crio-2962525dd254d46317c4567fa0a49ec13cd4e332fffe3746b973224fc339bd7c WatchSource:0}: Error finding container 2962525dd254d46317c4567fa0a49ec13cd4e332fffe3746b973224fc339bd7c: Status 404 returned error can't find the container with id 2962525dd254d46317c4567fa0a49ec13cd4e332fffe3746b973224fc339bd7c Oct 30 18:14:50.729184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.729161 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458q7\" (UniqueName: \"kubernetes.io/projected/34388c06-6a2e-4b53-b2e9-548a99e2535f-kube-api-access-458q7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.729184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:50.729185 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34388c06-6a2e-4b53-b2e9-548a99e2535f-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.108644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.108613 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:51.108750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.108662 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:14:51.390970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.390929 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" Oct 30 18:14:51.391145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.390929 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz" event={"ID":"34388c06-6a2e-4b53-b2e9-548a99e2535f","Type":"ContainerDied","Data":"24144646b954dba711e3e39e905ffd923e194e498399b2f173db067e795af714"} Oct 30 18:14:51.391145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.391060 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24144646b954dba711e3e39e905ffd923e194e498399b2f173db067e795af714" Oct 30 18:14:51.392661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.392634 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" event={"ID":"d36f3041-677f-4970-b8bb-a3f5bc19f167","Type":"ContainerStarted","Data":"d1a308a13bc9d618a06095a9147e2e941f96e939add93d29d73c10c94d9986d7"} Oct 30 18:14:51.392792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.392683 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" event={"ID":"d36f3041-677f-4970-b8bb-a3f5bc19f167","Type":"ContainerStarted","Data":"2962525dd254d46317c4567fa0a49ec13cd4e332fffe3746b973224fc339bd7c"} Oct 30 18:14:51.433700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.433657 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" podStartSLOduration=2.433645724 podStartE2EDuration="2.433645724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:51.426148377 +0000 UTC m=+988.250542670" watchObservedRunningTime="2026-10-30 18:14:51.433645724 +0000 UTC m=+988.258040015" Oct 30 18:14:51.459844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:51.459140 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:52.108022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:52.107937 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:52.108022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:52.108000 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:14:52.401497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:52.401461 4582 generic.go:356] "Generic (PLEG): container finished" podID="d36f3041-677f-4970-b8bb-a3f5bc19f167" containerID="d1a308a13bc9d618a06095a9147e2e941f96e939add93d29d73c10c94d9986d7" exitCode=0 Oct 30 18:14:52.401652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:52.401544 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" event={"ID":"d36f3041-677f-4970-b8bb-a3f5bc19f167","Type":"ContainerDied","Data":"d1a308a13bc9d618a06095a9147e2e941f96e939add93d29d73c10c94d9986d7"} Oct 30 18:14:53.108021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.107987 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:53.108409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.108039 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:14:53.187640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.187603 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:53.575283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.575249 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:53.687622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.687542 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36f3041-677f-4970-b8bb-a3f5bc19f167-kube-api-access-qq7bx\" (UniqueName: \"kubernetes.io/projected/d36f3041-677f-4970-b8bb-a3f5bc19f167-kube-api-access-qq7bx\") pod \"d36f3041-677f-4970-b8bb-a3f5bc19f167\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " Oct 30 18:14:53.687791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.687767 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-openshift-ca\") pod \"d36f3041-677f-4970-b8bb-a3f5bc19f167\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " Oct 30 18:14:53.687943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.687883 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-xks-ca\") pod \"d36f3041-677f-4970-b8bb-a3f5bc19f167\" (UID: \"d36f3041-677f-4970-b8bb-a3f5bc19f167\") " Oct 30 18:14:53.688110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.688086 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-openshift-ca" pod "d36f3041-677f-4970-b8bb-a3f5bc19f167" (UID: "d36f3041-677f-4970-b8bb-a3f5bc19f167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:53.688472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.688448 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:53.689680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.689655 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36f3041-677f-4970-b8bb-a3f5bc19f167-kube-api-access-qq7bx" pod "d36f3041-677f-4970-b8bb-a3f5bc19f167" (UID: "d36f3041-677f-4970-b8bb-a3f5bc19f167"). InnerVolumeSpecName "kube-api-access-qq7bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:53.689812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.689791 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-xks-ca" pod "d36f3041-677f-4970-b8bb-a3f5bc19f167" (UID: "d36f3041-677f-4970-b8bb-a3f5bc19f167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:53.789993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.789960 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq7bx\" (UniqueName: \"kubernetes.io/projected/d36f3041-677f-4970-b8bb-a3f5bc19f167-kube-api-access-qq7bx\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:53.789993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:53.789992 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f3041-677f-4970-b8bb-a3f5bc19f167-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:54.108444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.108361 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:54.108444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.108412 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:14:54.415912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.415877 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" Oct 30 18:14:54.416100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.415926 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq" event={"ID":"d36f3041-677f-4970-b8bb-a3f5bc19f167","Type":"ContainerDied","Data":"2962525dd254d46317c4567fa0a49ec13cd4e332fffe3746b973224fc339bd7c"} Oct 30 18:14:54.416100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.415978 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2962525dd254d46317c4567fa0a49ec13cd4e332fffe3746b973224fc339bd7c" Oct 30 18:14:54.565874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.565841 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl"] Oct 30 18:14:54.566379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.566366 4582 cpu_manager.go:414] "containerMap: removing container" podUID="34388c06-6a2e-4b53-b2e9-548a99e2535f" containerName="revoke-keys" Oct 30 18:14:54.566433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.566381 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34388c06-6a2e-4b53-b2e9-548a99e2535f" containerName="revoke-keys" Oct 30 18:14:54.566433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.566410 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d36f3041-677f-4970-b8bb-a3f5bc19f167" containerName="revoke-keys" Oct 30 18:14:54.566433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.566416 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36f3041-677f-4970-b8bb-a3f5bc19f167" containerName="revoke-keys" Oct 30 18:14:54.566525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.566516 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36f3041-677f-4970-b8bb-a3f5bc19f167" containerName="revoke-keys" Oct 30 18:14:54.566561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.566533 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="34388c06-6a2e-4b53-b2e9-548a99e2535f" containerName="revoke-keys" Oct 30 18:14:54.570050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.570026 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.584357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.584336 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:14:54.584545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.584513 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:54.588320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.588297 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl"] Oct 30 18:14:54.592957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.592934 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:14:54.592957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.592935 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:14:54.702590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.702495 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.702590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.702558 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgpf\" (UniqueName: \"kubernetes.io/projected/4f090e2d-6193-4bd2-975c-8468554fbaa4-kube-api-access-7cgpf\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.702805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.702676 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.804470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.804439 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.804640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.804480 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgpf\" (UniqueName: \"kubernetes.io/projected/4f090e2d-6193-4bd2-975c-8468554fbaa4-kube-api-access-7cgpf\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.804640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.804519 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.805358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.805325 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.807064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.807031 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.822094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.822068 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgpf\" (UniqueName: \"kubernetes.io/projected/4f090e2d-6193-4bd2-975c-8468554fbaa4-kube-api-access-7cgpf\") pod \"maas-api-revoke-sub-1bdf91da-bb5vl\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:54.883846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:54.883793 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:55.113325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:55.113257 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:55.113734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:55.113715 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:55.114592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:55.114568 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 18:14:55.119743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:55.119706 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:55.205892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:55.199870 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl"] Oct 30 18:14:55.423968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:55.423924 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" event={"ID":"4f090e2d-6193-4bd2-975c-8468554fbaa4","Type":"ContainerStarted","Data":"96732eae6359b16912b7eca2f03cd9b939ffda9e3eea70591a6c2e42a2aef7a3"} Oct 30 18:14:56.432306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:56.432272 4582 generic.go:356] "Generic (PLEG): container finished" podID="4f090e2d-6193-4bd2-975c-8468554fbaa4" containerID="05d822a86f0403bc7268c7d9178a2ec5866dae1036e0f215a1f3e4fef5a848df" exitCode=0 Oct 30 18:14:56.432735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:56.432349 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" event={"ID":"4f090e2d-6193-4bd2-975c-8468554fbaa4","Type":"ContainerDied","Data":"05d822a86f0403bc7268c7d9178a2ec5866dae1036e0f215a1f3e4fef5a848df"} Oct 30 18:14:56.885843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:56.885784 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:57.634641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.634296 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:57.765426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.765339 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-xks-ca\") pod \"4f090e2d-6193-4bd2-975c-8468554fbaa4\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " Oct 30 18:14:57.784398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.778175 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f090e2d-6193-4bd2-975c-8468554fbaa4-kube-api-access-7cgpf\" (UniqueName: \"kubernetes.io/projected/4f090e2d-6193-4bd2-975c-8468554fbaa4-kube-api-access-7cgpf\") pod \"4f090e2d-6193-4bd2-975c-8468554fbaa4\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " Oct 30 18:14:57.784398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.778356 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-openshift-ca\") pod \"4f090e2d-6193-4bd2-975c-8468554fbaa4\" (UID: \"4f090e2d-6193-4bd2-975c-8468554fbaa4\") " Oct 30 18:14:57.784398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.784047 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-openshift-ca" pod "4f090e2d-6193-4bd2-975c-8468554fbaa4" (UID: "4f090e2d-6193-4bd2-975c-8468554fbaa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:57.869432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.869374 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f090e2d-6193-4bd2-975c-8468554fbaa4-kube-api-access-7cgpf" pod "4f090e2d-6193-4bd2-975c-8468554fbaa4" (UID: "4f090e2d-6193-4bd2-975c-8468554fbaa4"). InnerVolumeSpecName "kube-api-access-7cgpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:57.945851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.917166 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cgpf\" (UniqueName: \"kubernetes.io/projected/4f090e2d-6193-4bd2-975c-8468554fbaa4-kube-api-access-7cgpf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:57.961188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:57.961128 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:58.033033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.032948 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-xks-ca" pod "4f090e2d-6193-4bd2-975c-8468554fbaa4" (UID: "4f090e2d-6193-4bd2-975c-8468554fbaa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:58.065086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.065055 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f090e2d-6193-4bd2-975c-8468554fbaa4-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:14:58.472844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.472680 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" event={"ID":"4f090e2d-6193-4bd2-975c-8468554fbaa4","Type":"ContainerDied","Data":"96732eae6359b16912b7eca2f03cd9b939ffda9e3eea70591a6c2e42a2aef7a3"} Oct 30 18:14:58.472844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.472742 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96732eae6359b16912b7eca2f03cd9b939ffda9e3eea70591a6c2e42a2aef7a3" Oct 30 18:14:58.472844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.472815 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl" Oct 30 18:14:58.748901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.747925 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:14:58.825074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.825027 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:58.825970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.825329 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:14:58.949977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:58.949910 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:59.158844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.158074 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:59.618815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.618775 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:59.748957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.748911 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:59.749982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.749959 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:14:59.761566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.760886 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:59.826938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.826895 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:14:59.828668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.828643 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:14:59.896013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.895906 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:59.896664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.896635 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:59.929844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.898049 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:15:00.165898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:14:59.977953 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:00.165898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.161699 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:15:00.171927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.171896 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb"] Oct 30 18:15:00.189096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.189066 4582 cpu_manager.go:414] "containerMap: removing container" podUID="4f090e2d-6193-4bd2-975c-8468554fbaa4" containerName="revoke-keys" Oct 30 18:15:00.189575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.189506 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f090e2d-6193-4bd2-975c-8468554fbaa4" containerName="revoke-keys" Oct 30 18:15:00.189987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.189966 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f090e2d-6193-4bd2-975c-8468554fbaa4" containerName="revoke-keys" Oct 30 18:15:00.374331 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.374301 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.418900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.402219 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:15:00.448099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.448066 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:15:00.448462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.448403 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 18:15:00.477460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.477436 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:15:00.498843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.485900 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl"] Oct 30 18:15:00.498843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.495989 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:00.498843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.496791 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.498843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.496945 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpq6w\" (UniqueName: \"kubernetes.io/projected/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-kube-api-access-wpq6w\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.648258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.627894 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:00.712985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.700724 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.712985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.700813 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpq6w\" (UniqueName: \"kubernetes.io/projected/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-kube-api-access-wpq6w\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.758844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.743865 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.811861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.784230 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:00.891876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.886510 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:15:00.891876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.886567 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg"] Oct 30 18:15:00.906690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.906643 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:00.907198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.907175 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:00.932029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.932001 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw"] Oct 30 18:15:00.934443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.934421 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:00.934980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.932569 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpq6w\" (UniqueName: \"kubernetes.io/projected/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-kube-api-access-wpq6w\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.939289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.939267 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:00.942033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.942010 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:15:00.942133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.942114 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:00.942204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.942190 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:15:00.942272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.942261 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:15:00.942330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.942284 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:15:00.942379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.942342 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:15:00.947247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.944135 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:00.949293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.949266 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:15:00.953631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.953605 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb"] Oct 30 18:15:00.958975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.958908 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl"] Oct 30 18:15:00.965010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.964945 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg"] Oct 30 18:15:00.979985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.979875 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw"] Oct 30 18:15:00.998757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.998730 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8704aa-cb73-4c59-a639-4582df054e7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-nrwzl\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:00.998966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:00.998946 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvfs\" (UniqueName: \"kubernetes.io/projected/9a8704aa-cb73-4c59-a639-4582df054e7c-kube-api-access-9jvfs\") pod \"maas-api-key-cleanup-29889735-nrwzl\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:01.100288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.100252 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8704aa-cb73-4c59-a639-4582df054e7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-nrwzl\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:01.100485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.100295 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvfs\" (UniqueName: \"kubernetes.io/projected/9a8704aa-cb73-4c59-a639-4582df054e7c-kube-api-access-9jvfs\") pod \"maas-api-key-cleanup-29889735-nrwzl\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:01.100485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.100434 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43a9bd6-dd57-4327-830c-64ed31ea8da4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:01.101116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.101093 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8704aa-cb73-4c59-a639-4582df054e7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-nrwzl\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:01.101222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.101167 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqvf\" (UniqueName: \"kubernetes.io/projected/b43a9bd6-dd57-4327-830c-64ed31ea8da4-kube-api-access-glqvf\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:01.101272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.101245 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b9483c-0a8e-4e34-820b-f85f5e0563e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:01.101570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.101545 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslrd\" (UniqueName: \"kubernetes.io/projected/62b9483c-0a8e-4e34-820b-f85f5e0563e1-kube-api-access-qslrd\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:01.119376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.119327 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvfs\" (UniqueName: \"kubernetes.io/projected/9a8704aa-cb73-4c59-a639-4582df054e7c-kube-api-access-9jvfs\") pod \"maas-api-key-cleanup-29889735-nrwzl\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:01.178197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.178158 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:01.203328 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.203293 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslrd\" (UniqueName: \"kubernetes.io/projected/62b9483c-0a8e-4e34-820b-f85f5e0563e1-kube-api-access-qslrd\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:01.203503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.203464 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43a9bd6-dd57-4327-830c-64ed31ea8da4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:01.203570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.203511 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqvf\" (UniqueName: \"kubernetes.io/projected/b43a9bd6-dd57-4327-830c-64ed31ea8da4-kube-api-access-glqvf\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:01.203570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.203538 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b9483c-0a8e-4e34-820b-f85f5e0563e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:01.204323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.204300 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b9483c-0a8e-4e34-820b-f85f5e0563e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:01.204468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.204327 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43a9bd6-dd57-4327-830c-64ed31ea8da4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:01.219996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.219917 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslrd\" (UniqueName: \"kubernetes.io/projected/62b9483c-0a8e-4e34-820b-f85f5e0563e1-kube-api-access-qslrd\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:01.220102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.220029 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqvf\" (UniqueName: \"kubernetes.io/projected/b43a9bd6-dd57-4327-830c-64ed31ea8da4-kube-api-access-glqvf\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:01.263249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.263221 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:01.274760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.274718 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb"] Oct 30 18:15:01.280023 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:01.279992 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3f253e_6c97_418c_8cb5_6e95c25e3c1d.slice/crio-40e8f39f9e87ff20d8b2a8d5f8ea94c0d7e325e1ca7d3e92040e955ad1088e36 WatchSource:0}: Error finding container 40e8f39f9e87ff20d8b2a8d5f8ea94c0d7e325e1ca7d3e92040e955ad1088e36: Status 404 returned error can't find the container with id 40e8f39f9e87ff20d8b2a8d5f8ea94c0d7e325e1ca7d3e92040e955ad1088e36 Oct 30 18:15:01.319814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.319786 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:01.335920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.335893 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:01.573744 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:01.573696 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8704aa_cb73_4c59_a639_4582df054e7c.slice/crio-cf4623ce70b8271839ba6f24f0cc7e05d09724f58acfc65a6f0085474e830ddf WatchSource:0}: Error finding container cf4623ce70b8271839ba6f24f0cc7e05d09724f58acfc65a6f0085474e830ddf: Status 404 returned error can't find the container with id cf4623ce70b8271839ba6f24f0cc7e05d09724f58acfc65a6f0085474e830ddf Oct 30 18:15:01.576135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.576102 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl"] Oct 30 18:15:01.714113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.714089 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg"] Oct 30 18:15:01.716313 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:01.716281 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b9483c_0a8e_4e34_820b_f85f5e0563e1.slice/crio-4f64f856ef5fc6daf88cda8b6dbbc13847479459c2703ebd10e4fc7313168234 WatchSource:0}: Error finding container 4f64f856ef5fc6daf88cda8b6dbbc13847479459c2703ebd10e4fc7313168234: Status 404 returned error can't find the container with id 4f64f856ef5fc6daf88cda8b6dbbc13847479459c2703ebd10e4fc7313168234 Oct 30 18:15:01.751872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.751844 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw"] Oct 30 18:15:01.755991 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:01.755964 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43a9bd6_dd57_4327_830c_64ed31ea8da4.slice/crio-b683f1b529ca8274a80524e7f50cc63890f69200222ec22d2393fbdf466be285 WatchSource:0}: Error finding container b683f1b529ca8274a80524e7f50cc63890f69200222ec22d2393fbdf466be285: Status 404 returned error can't find the container with id b683f1b529ca8274a80524e7f50cc63890f69200222ec22d2393fbdf466be285 Oct 30 18:15:01.942162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.942121 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" event={"ID":"62b9483c-0a8e-4e34-820b-f85f5e0563e1","Type":"ContainerStarted","Data":"4f64f856ef5fc6daf88cda8b6dbbc13847479459c2703ebd10e4fc7313168234"} Oct 30 18:15:01.944073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.944046 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" event={"ID":"9a8704aa-cb73-4c59-a639-4582df054e7c","Type":"ContainerStarted","Data":"0e659e46d56af689ce6056f6eb8999495cb23054fbf4da3cc3141e055801d9c1"} Oct 30 18:15:01.944205 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.944095 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" event={"ID":"9a8704aa-cb73-4c59-a639-4582df054e7c","Type":"ContainerStarted","Data":"cf4623ce70b8271839ba6f24f0cc7e05d09724f58acfc65a6f0085474e830ddf"} Oct 30 18:15:01.945872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.945846 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" event={"ID":"b43a9bd6-dd57-4327-830c-64ed31ea8da4","Type":"ContainerStarted","Data":"b683f1b529ca8274a80524e7f50cc63890f69200222ec22d2393fbdf466be285"} Oct 30 18:15:01.947775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.947739 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" event={"ID":"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d","Type":"ContainerStarted","Data":"41dfa2ecc0d1100fbcdab021ee637dbf062f1f9b180072f6178467517788b74f"} Oct 30 18:15:01.947911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.947787 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" event={"ID":"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d","Type":"ContainerStarted","Data":"40e8f39f9e87ff20d8b2a8d5f8ea94c0d7e325e1ca7d3e92040e955ad1088e36"} Oct 30 18:15:01.977513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:01.977472 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" podStartSLOduration=1.977457233 podStartE2EDuration="1.977457233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:01.975828825 +0000 UTC m=+998.800223128" watchObservedRunningTime="2026-10-30 18:15:01.977457233 +0000 UTC m=+998.801851525" Oct 30 18:15:02.009056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.008980 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" podStartSLOduration=2.008968161 podStartE2EDuration="2.008968161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:02.007171581 +0000 UTC m=+998.831565871" watchObservedRunningTime="2026-10-30 18:15:02.008968161 +0000 UTC m=+998.833362452" Oct 30 18:15:02.956994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.956915 4582 generic.go:356] "Generic (PLEG): container finished" podID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerID="41dfa2ecc0d1100fbcdab021ee637dbf062f1f9b180072f6178467517788b74f" exitCode=60 Oct 30 18:15:02.957369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.957002 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" event={"ID":"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d","Type":"ContainerDied","Data":"41dfa2ecc0d1100fbcdab021ee637dbf062f1f9b180072f6178467517788b74f"} Oct 30 18:15:02.957369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.957217 4582 scope.go:122] "RemoveContainer" containerID="41dfa2ecc0d1100fbcdab021ee637dbf062f1f9b180072f6178467517788b74f" Oct 30 18:15:02.959029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.959007 4582 generic.go:356] "Generic (PLEG): container finished" podID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerID="712be26082f3837fc69d642450e43d1342717dca751ffb3b1cd8cd479158cec9" exitCode=60 Oct 30 18:15:02.959139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.959083 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" event={"ID":"62b9483c-0a8e-4e34-820b-f85f5e0563e1","Type":"ContainerDied","Data":"712be26082f3837fc69d642450e43d1342717dca751ffb3b1cd8cd479158cec9"} Oct 30 18:15:02.959289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.959270 4582 scope.go:122] "RemoveContainer" containerID="712be26082f3837fc69d642450e43d1342717dca751ffb3b1cd8cd479158cec9" Oct 30 18:15:02.961057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.960977 4582 generic.go:356] "Generic (PLEG): container finished" podID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerID="0e659e46d56af689ce6056f6eb8999495cb23054fbf4da3cc3141e055801d9c1" exitCode=60 Oct 30 18:15:02.961119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.961058 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" event={"ID":"9a8704aa-cb73-4c59-a639-4582df054e7c","Type":"ContainerDied","Data":"0e659e46d56af689ce6056f6eb8999495cb23054fbf4da3cc3141e055801d9c1"} Oct 30 18:15:02.961275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.961260 4582 scope.go:122] "RemoveContainer" containerID="0e659e46d56af689ce6056f6eb8999495cb23054fbf4da3cc3141e055801d9c1" Oct 30 18:15:02.962624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.962603 4582 generic.go:356] "Generic (PLEG): container finished" podID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerID="0cd3a0d75ebffcf8fe7a249192d0c34933005f05ccc78fcd65301a445f28934d" exitCode=60 Oct 30 18:15:02.962734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.962645 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" event={"ID":"b43a9bd6-dd57-4327-830c-64ed31ea8da4","Type":"ContainerDied","Data":"0cd3a0d75ebffcf8fe7a249192d0c34933005f05ccc78fcd65301a445f28934d"} Oct 30 18:15:02.962900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:02.962886 4582 scope.go:122] "RemoveContainer" containerID="0cd3a0d75ebffcf8fe7a249192d0c34933005f05ccc78fcd65301a445f28934d" Oct 30 18:15:03.366410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.366387 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:03.971244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.971215 4582 generic.go:356] "Generic (PLEG): container finished" podID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerID="b755ad4653c164b0bf866b11abfc128e67cad954154b7744164831233db6c642" exitCode=60 Oct 30 18:15:03.971596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.971286 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" event={"ID":"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d","Type":"ContainerDied","Data":"b755ad4653c164b0bf866b11abfc128e67cad954154b7744164831233db6c642"} Oct 30 18:15:03.971596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.971374 4582 scope.go:122] "RemoveContainer" containerID="41dfa2ecc0d1100fbcdab021ee637dbf062f1f9b180072f6178467517788b74f" Oct 30 18:15:03.971596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.971584 4582 scope.go:122] "RemoveContainer" containerID="b755ad4653c164b0bf866b11abfc128e67cad954154b7744164831233db6c642" Oct 30 18:15:03.971919 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:03.971888 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb_odh-ai-gateway-infra(7a3f253e-6c97-418c-8cb5-6e95c25e3c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" Oct 30 18:15:03.973362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.973337 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" event={"ID":"62b9483c-0a8e-4e34-820b-f85f5e0563e1","Type":"ContainerStarted","Data":"67b83cfd424addeac9a408d5048db4834d7e0de18053cd4919c0507d08d1ac10"} Oct 30 18:15:03.975217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.975198 4582 generic.go:356] "Generic (PLEG): container finished" podID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerID="915cc355d394fbfee31345c8dbe21d06e76710a9de3fce7d9427e57d7ea4373a" exitCode=60 Oct 30 18:15:03.975309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.975283 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" event={"ID":"9a8704aa-cb73-4c59-a639-4582df054e7c","Type":"ContainerDied","Data":"915cc355d394fbfee31345c8dbe21d06e76710a9de3fce7d9427e57d7ea4373a"} Oct 30 18:15:03.975495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.975478 4582 scope.go:122] "RemoveContainer" containerID="915cc355d394fbfee31345c8dbe21d06e76710a9de3fce7d9427e57d7ea4373a" Oct 30 18:15:03.975747 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:03.975726 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-nrwzl_odh-ai-gateway-infra(9a8704aa-cb73-4c59-a639-4582df054e7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" Oct 30 18:15:03.977185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.977165 4582 generic.go:356] "Generic (PLEG): container finished" podID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerID="0195aad76dff224eba89467fdea9e0c1f91821137763b837586041798700e3e3" exitCode=60 Oct 30 18:15:03.977265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.977198 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" event={"ID":"b43a9bd6-dd57-4327-830c-64ed31ea8da4","Type":"ContainerDied","Data":"0195aad76dff224eba89467fdea9e0c1f91821137763b837586041798700e3e3"} Oct 30 18:15:03.977386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:03.977372 4582 scope.go:122] "RemoveContainer" containerID="0195aad76dff224eba89467fdea9e0c1f91821137763b837586041798700e3e3" Oct 30 18:15:03.977559 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:03.977544 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw_odh-ai-gateway-infra(b43a9bd6-dd57-4327-830c-64ed31ea8da4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" Oct 30 18:15:04.028725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.028703 4582 scope.go:122] "RemoveContainer" containerID="0e659e46d56af689ce6056f6eb8999495cb23054fbf4da3cc3141e055801d9c1" Oct 30 18:15:04.039426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.039385 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" podStartSLOduration=4.039372656 podStartE2EDuration="4.039372656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:04.03448703 +0000 UTC m=+1000.858881324" watchObservedRunningTime="2026-10-30 18:15:04.039372656 +0000 UTC m=+1000.863766945" Oct 30 18:15:04.044406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.044384 4582 scope.go:122] "RemoveContainer" containerID="0cd3a0d75ebffcf8fe7a249192d0c34933005f05ccc78fcd65301a445f28934d" Oct 30 18:15:04.647568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.647532 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:04.985619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.985541 4582 generic.go:356] "Generic (PLEG): container finished" podID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerID="67b83cfd424addeac9a408d5048db4834d7e0de18053cd4919c0507d08d1ac10" exitCode=60 Oct 30 18:15:04.986132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.985625 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" event={"ID":"62b9483c-0a8e-4e34-820b-f85f5e0563e1","Type":"ContainerDied","Data":"67b83cfd424addeac9a408d5048db4834d7e0de18053cd4919c0507d08d1ac10"} Oct 30 18:15:04.986132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.985693 4582 scope.go:122] "RemoveContainer" containerID="712be26082f3837fc69d642450e43d1342717dca751ffb3b1cd8cd479158cec9" Oct 30 18:15:04.986132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.985902 4582 scope.go:122] "RemoveContainer" containerID="67b83cfd424addeac9a408d5048db4834d7e0de18053cd4919c0507d08d1ac10" Oct 30 18:15:04.986298 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:04.986193 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg_odh-ai-gateway-infra(62b9483c-0a8e-4e34-820b-f85f5e0563e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" Oct 30 18:15:04.987691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.987672 4582 scope.go:122] "RemoveContainer" containerID="915cc355d394fbfee31345c8dbe21d06e76710a9de3fce7d9427e57d7ea4373a" Oct 30 18:15:04.988021 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:04.987983 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-nrwzl_odh-ai-gateway-infra(9a8704aa-cb73-4c59-a639-4582df054e7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" Oct 30 18:15:04.990375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.990351 4582 scope.go:122] "RemoveContainer" containerID="0195aad76dff224eba89467fdea9e0c1f91821137763b837586041798700e3e3" Oct 30 18:15:04.990624 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:04.990601 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw_odh-ai-gateway-infra(b43a9bd6-dd57-4327-830c-64ed31ea8da4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" Oct 30 18:15:04.994169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:04.992692 4582 scope.go:122] "RemoveContainer" containerID="b755ad4653c164b0bf866b11abfc128e67cad954154b7744164831233db6c642" Oct 30 18:15:04.994169 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:04.992935 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb_odh-ai-gateway-infra(7a3f253e-6c97-418c-8cb5-6e95c25e3c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" Oct 30 18:15:05.041026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.040996 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp"] Oct 30 18:15:05.046763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.046742 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.059554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.059530 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:05.061779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.061752 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp"] Oct 30 18:15:05.157965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.157935 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.158372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.158341 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.158515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.158471 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7ts\" (UniqueName: \"kubernetes.io/projected/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-kube-api-access-fv7ts\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.178794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.178766 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:05.260099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.260010 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.260289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.260177 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.260289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.260221 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7ts\" (UniqueName: \"kubernetes.io/projected/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-kube-api-access-fv7ts\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.260781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.260749 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.262737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.262712 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.276985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.276941 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7ts\" (UniqueName: \"kubernetes.io/projected/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-kube-api-access-fv7ts\") pod \"maas-api-revoke-sub-feb96a6a-kkxvp\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.361111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.361079 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:05.661544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:05.661520 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp"] Oct 30 18:15:05.664219 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:05.664191 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481c1c02_3b8c_4a58_a62a_ab5a7579ffe1.slice/crio-3cecb8c96beb94c1cc2070c6e3c48cf4a84fd1b0c98a456e2a2e4c4e768d85c8 WatchSource:0}: Error finding container 3cecb8c96beb94c1cc2070c6e3c48cf4a84fd1b0c98a456e2a2e4c4e768d85c8: Status 404 returned error can't find the container with id 3cecb8c96beb94c1cc2070c6e3c48cf4a84fd1b0c98a456e2a2e4c4e768d85c8 Oct 30 18:15:06.009058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.009016 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" event={"ID":"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1","Type":"ContainerStarted","Data":"7b5e2bc5c52096d10bf34c36e5ca1310a8ec7fd693c19c51d08bb7e49634d5fa"} Oct 30 18:15:06.009536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.009103 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" event={"ID":"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1","Type":"ContainerStarted","Data":"3cecb8c96beb94c1cc2070c6e3c48cf4a84fd1b0c98a456e2a2e4c4e768d85c8"} Oct 30 18:15:06.010739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.010721 4582 scope.go:122] "RemoveContainer" containerID="67b83cfd424addeac9a408d5048db4834d7e0de18053cd4919c0507d08d1ac10" Oct 30 18:15:06.010991 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:06.010972 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg_odh-ai-gateway-infra(62b9483c-0a8e-4e34-820b-f85f5e0563e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" Oct 30 18:15:06.045333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.045258 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" podStartSLOduration=1.045240535 podStartE2EDuration="1.045240535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:06.043068505 +0000 UTC m=+1002.867462809" watchObservedRunningTime="2026-10-30 18:15:06.045240535 +0000 UTC m=+1002.869634827" Oct 30 18:15:06.225056 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:06.224978 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:15:06.705049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.705014 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w"] Oct 30 18:15:06.709616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.709595 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.731068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.726244 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:06.731068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.726749 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w"] Oct 30 18:15:06.780396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.780365 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfkm\" (UniqueName: \"kubernetes.io/projected/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-kube-api-access-tbfkm\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.780639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.780608 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.780931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.780703 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.882754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.882714 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.882967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.882772 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.882967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.882866 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfkm\" (UniqueName: \"kubernetes.io/projected/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-kube-api-access-tbfkm\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.883633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.883602 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.885224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.885193 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:06.899551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:06.899524 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfkm\" (UniqueName: \"kubernetes.io/projected/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-kube-api-access-tbfkm\") pod \"maas-api-revoke-sub-ba370bee-lcz8w\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:07.020045 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:07.020004 4582 generic.go:356] "Generic (PLEG): container finished" podID="481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" containerID="7b5e2bc5c52096d10bf34c36e5ca1310a8ec7fd693c19c51d08bb7e49634d5fa" exitCode=0 Oct 30 18:15:07.020440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:07.020081 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" event={"ID":"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1","Type":"ContainerDied","Data":"7b5e2bc5c52096d10bf34c36e5ca1310a8ec7fd693c19c51d08bb7e49634d5fa"} Oct 30 18:15:07.024298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:07.024276 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:07.329049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:07.329018 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w"] Oct 30 18:15:07.330740 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:07.330712 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5203f7c_de3e_4d11_ad89_b54ec1f2a21e.slice/crio-617a085cced9a9674137f9d29431f7d6e4e4043120b35ac0869c65c8bc34ca19 WatchSource:0}: Error finding container 617a085cced9a9674137f9d29431f7d6e4e4043120b35ac0869c65c8bc34ca19: Status 404 returned error can't find the container with id 617a085cced9a9674137f9d29431f7d6e4e4043120b35ac0869c65c8bc34ca19 Oct 30 18:15:08.029710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.029614 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" event={"ID":"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e","Type":"ContainerStarted","Data":"337499cf1c8bf76f97906af6cf791527c83aea8653c75e0410dd65000f867de3"} Oct 30 18:15:08.029710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.029698 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" event={"ID":"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e","Type":"ContainerStarted","Data":"617a085cced9a9674137f9d29431f7d6e4e4043120b35ac0869c65c8bc34ca19"} Oct 30 18:15:08.063975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.063909 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" podStartSLOduration=2.063891657 podStartE2EDuration="2.063891657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:08.062122146 +0000 UTC m=+1004.886516438" watchObservedRunningTime="2026-10-30 18:15:08.063891657 +0000 UTC m=+1004.888285950" Oct 30 18:15:08.183197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.183170 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:08.204252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.204219 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-kube-api-access-fv7ts\" (UniqueName: \"kubernetes.io/projected/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-kube-api-access-fv7ts\") pod \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " Oct 30 18:15:08.204386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.204288 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-xks-ca\") pod \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " Oct 30 18:15:08.204434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.204387 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-openshift-ca\") pod \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\" (UID: \"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1\") " Oct 30 18:15:08.204738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.204714 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-openshift-ca" pod "481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" (UID: "481c1c02-3b8c-4a58-a62a-ab5a7579ffe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:08.204919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.204898 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:08.206203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.206175 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-xks-ca" pod "481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" (UID: "481c1c02-3b8c-4a58-a62a-ab5a7579ffe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:08.206301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.206281 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-kube-api-access-fv7ts" pod "481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" (UID: "481c1c02-3b8c-4a58-a62a-ab5a7579ffe1"). InnerVolumeSpecName "kube-api-access-fv7ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:08.306408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.306333 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv7ts\" (UniqueName: \"kubernetes.io/projected/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-kube-api-access-fv7ts\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:08.306408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.306361 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:08.348689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:08.348658 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:09.038400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:09.038372 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" Oct 30 18:15:09.038800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:09.038407 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp" event={"ID":"481c1c02-3b8c-4a58-a62a-ab5a7579ffe1","Type":"ContainerDied","Data":"3cecb8c96beb94c1cc2070c6e3c48cf4a84fd1b0c98a456e2a2e4c4e768d85c8"} Oct 30 18:15:09.038800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:09.038450 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cecb8c96beb94c1cc2070c6e3c48cf4a84fd1b0c98a456e2a2e4c4e768d85c8" Oct 30 18:15:09.040192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:09.040162 4582 generic.go:356] "Generic (PLEG): container finished" podID="c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" containerID="337499cf1c8bf76f97906af6cf791527c83aea8653c75e0410dd65000f867de3" exitCode=0 Oct 30 18:15:09.040328 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:09.040231 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" event={"ID":"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e","Type":"ContainerDied","Data":"337499cf1c8bf76f97906af6cf791527c83aea8653c75e0410dd65000f867de3"} Oct 30 18:15:10.035254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.035222 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:10.221884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.221859 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:10.330062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.329977 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-xks-ca\") pod \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " Oct 30 18:15:10.330244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.330098 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-kube-api-access-tbfkm\" (UniqueName: \"kubernetes.io/projected/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-kube-api-access-tbfkm\") pod \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " Oct 30 18:15:10.330318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.330252 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-openshift-ca\") pod \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\" (UID: \"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e\") " Oct 30 18:15:10.330603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.330579 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-openshift-ca" pod "c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" (UID: "c5203f7c-de3e-4d11-ad89-b54ec1f2a21e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:10.330784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.330768 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.332204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.332180 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-xks-ca" pod "c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" (UID: "c5203f7c-de3e-4d11-ad89-b54ec1f2a21e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:10.332295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.332264 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-kube-api-access-tbfkm" pod "c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" (UID: "c5203f7c-de3e-4d11-ad89-b54ec1f2a21e"). InnerVolumeSpecName "kube-api-access-tbfkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:10.432509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.432480 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.432509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:10.432506 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfkm\" (UniqueName: \"kubernetes.io/projected/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e-kube-api-access-tbfkm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:11.056040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:11.056010 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" Oct 30 18:15:11.056185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:11.056007 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w" event={"ID":"c5203f7c-de3e-4d11-ad89-b54ec1f2a21e","Type":"ContainerDied","Data":"617a085cced9a9674137f9d29431f7d6e4e4043120b35ac0869c65c8bc34ca19"} Oct 30 18:15:11.056185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:11.056157 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="617a085cced9a9674137f9d29431f7d6e4e4043120b35ac0869c65c8bc34ca19" Oct 30 18:15:11.633540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:11.633509 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:13.234990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:13.234955 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:13.336555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:13.336523 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:14.852806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:14.852771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:15.293274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:15.293234 4582 scope.go:122] "RemoveContainer" containerID="67b83cfd424addeac9a408d5048db4834d7e0de18053cd4919c0507d08d1ac10" Oct 30 18:15:15.293423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:15.293337 4582 scope.go:122] "RemoveContainer" containerID="0195aad76dff224eba89467fdea9e0c1f91821137763b837586041798700e3e3" Oct 30 18:15:15.293459 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:15.293430 4582 scope.go:122] "RemoveContainer" containerID="915cc355d394fbfee31345c8dbe21d06e76710a9de3fce7d9427e57d7ea4373a" Oct 30 18:15:15.293513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:15.293472 4582 scope.go:122] "RemoveContainer" containerID="b755ad4653c164b0bf866b11abfc128e67cad954154b7744164831233db6c642" Oct 30 18:15:16.131889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:16.130189 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" event={"ID":"62b9483c-0a8e-4e34-820b-f85f5e0563e1","Type":"ContainerStarted","Data":"3f701874702b7186ea3bc2c9ff5eb02d8f886096079c830f414d146ab27eb31a"} Oct 30 18:15:16.152855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:16.150445 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" event={"ID":"9a8704aa-cb73-4c59-a639-4582df054e7c","Type":"ContainerStarted","Data":"ba01827b4716eb6a7d1b1456c0c4592ead859ee93ecdb77c634bb837587ae571"} Oct 30 18:15:16.168853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:16.167782 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" event={"ID":"b43a9bd6-dd57-4327-830c-64ed31ea8da4","Type":"ContainerStarted","Data":"6b3a0c43fd3b019a0c919ec8d19c84d2772ef9752e58c2ff5439f928212534ed"} Oct 30 18:15:16.201863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:16.199573 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" event={"ID":"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d","Type":"ContainerStarted","Data":"d1dfe7885b6c31421947e7337538a5a9bb6d98c3caa1b47cc0bd0078d4ebf3e7"} Oct 30 18:15:16.345933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:16.338635 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" podStartSLOduration=16.338615745 podStartE2EDuration="16.338615745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:16.316098513 +0000 UTC m=+1013.140492806" watchObservedRunningTime="2026-10-30 18:15:16.338615745 +0000 UTC m=+1013.163010038" Oct 30 18:15:17.278862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.277079 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg"] Oct 30 18:15:17.363884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.363843 4582 generic.go:356] "Generic (PLEG): container finished" podID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerID="ba01827b4716eb6a7d1b1456c0c4592ead859ee93ecdb77c634bb837587ae571" exitCode=60 Oct 30 18:15:17.382483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.382275 4582 generic.go:356] "Generic (PLEG): container finished" podID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerID="6b3a0c43fd3b019a0c919ec8d19c84d2772ef9752e58c2ff5439f928212534ed" exitCode=60 Oct 30 18:15:17.385270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.385157 4582 generic.go:356] "Generic (PLEG): container finished" podID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerID="d1dfe7885b6c31421947e7337538a5a9bb6d98c3caa1b47cc0bd0078d4ebf3e7" exitCode=60 Oct 30 18:15:17.404292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.403841 4582 generic.go:356] "Generic (PLEG): container finished" podID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerID="3f701874702b7186ea3bc2c9ff5eb02d8f886096079c830f414d146ab27eb31a" exitCode=60 Oct 30 18:15:17.427532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.427499 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" event={"ID":"9a8704aa-cb73-4c59-a639-4582df054e7c","Type":"ContainerDied","Data":"ba01827b4716eb6a7d1b1456c0c4592ead859ee93ecdb77c634bb837587ae571"} Oct 30 18:15:17.427787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.427763 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" event={"ID":"b43a9bd6-dd57-4327-830c-64ed31ea8da4","Type":"ContainerDied","Data":"6b3a0c43fd3b019a0c919ec8d19c84d2772ef9752e58c2ff5439f928212534ed"} Oct 30 18:15:17.432004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.431978 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" event={"ID":"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d","Type":"ContainerDied","Data":"d1dfe7885b6c31421947e7337538a5a9bb6d98c3caa1b47cc0bd0078d4ebf3e7"} Oct 30 18:15:17.432146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.432125 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" event={"ID":"62b9483c-0a8e-4e34-820b-f85f5e0563e1","Type":"ContainerDied","Data":"3f701874702b7186ea3bc2c9ff5eb02d8f886096079c830f414d146ab27eb31a"} Oct 30 18:15:17.432679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.432659 4582 scope.go:122] "RemoveContainer" containerID="d1dfe7885b6c31421947e7337538a5a9bb6d98c3caa1b47cc0bd0078d4ebf3e7" Oct 30 18:15:17.433175 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:17.433147 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb_odh-ai-gateway-infra(7a3f253e-6c97-418c-8cb5-6e95c25e3c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" Oct 30 18:15:17.450844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.433998 4582 scope.go:122] "RemoveContainer" containerID="ba01827b4716eb6a7d1b1456c0c4592ead859ee93ecdb77c634bb837587ae571" Oct 30 18:15:17.451355 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:17.451324 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-nrwzl_odh-ai-gateway-infra(9a8704aa-cb73-4c59-a639-4582df054e7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" Oct 30 18:15:17.453268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.452136 4582 scope.go:122] "RemoveContainer" containerID="915cc355d394fbfee31345c8dbe21d06e76710a9de3fce7d9427e57d7ea4373a" Oct 30 18:15:17.453546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.453528 4582 scope.go:122] "RemoveContainer" containerID="6b3a0c43fd3b019a0c919ec8d19c84d2772ef9752e58c2ff5439f928212534ed" Oct 30 18:15:17.453967 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:17.453944 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw_odh-ai-gateway-infra(b43a9bd6-dd57-4327-830c-64ed31ea8da4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" Oct 30 18:15:17.521555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.521520 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl"] Oct 30 18:15:17.659849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.655121 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.677846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.665423 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.731843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.717453 4582 scope.go:122] "RemoveContainer" containerID="0195aad76dff224eba89467fdea9e0c1f91821137763b837586041798700e3e3" Oct 30 18:15:17.797846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.761955 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.797846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.796038 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.951902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.926934 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:15:17.977842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.952588 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:15:17.978524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.970912 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.985850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:17.971895 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.045884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.045667 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 30 18:15:18.137426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.137386 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:18.137680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.137651 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.296105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.277089 4582 scope.go:122] "RemoveContainer" containerID="b755ad4653c164b0bf866b11abfc128e67cad954154b7744164831233db6c642" Oct 30 18:15:18.377276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.377240 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw"] Oct 30 18:15:18.843901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.837370 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.857208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.856585 4582 scope.go:122] "RemoveContainer" containerID="67b83cfd424addeac9a408d5048db4834d7e0de18053cd4919c0507d08d1ac10" Oct 30 18:15:18.866848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.866019 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:18.880854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.880369 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:18.880854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.880428 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:15:18.892847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:18.882170 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj" podUID="bad83681-b01e-416a-bd8a-fafae54c4f56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:19.039996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.011957 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:19.039996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.012137 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:19.039996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.013595 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:15:19.039996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.013662 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:15:19.190846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.173945 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:19.190846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.174019 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:19.221855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.203928 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:15:19.221855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.212145 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:15:19.221855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.213481 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:15:19.221855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.213563 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:15:19.243934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.243897 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 18:15:19.244226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.244197 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:19.263930 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.250119 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:15:19.425304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.425255 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:19.626969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.620353 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:19.766846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.751049 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:19.766846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.751482 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:19.856628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.856590 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:15:19.873705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.871156 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:19.946853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:19.946794 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b9483c-0a8e-4e34-820b-f85f5e0563e1-kube-api-access-qslrd\" (UniqueName: \"kubernetes.io/projected/62b9483c-0a8e-4e34-820b-f85f5e0563e1-kube-api-access-qslrd\") pod \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " Oct 30 18:15:20.002163 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.002068 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b9483c-0a8e-4e34-820b-f85f5e0563e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b9483c-0a8e-4e34-820b-f85f5e0563e1-maas-api-service-ca\") pod \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\" (UID: \"62b9483c-0a8e-4e34-820b-f85f5e0563e1\") " Oct 30 18:15:20.016087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.016044 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b9483c-0a8e-4e34-820b-f85f5e0563e1-maas-api-service-ca" pod "62b9483c-0a8e-4e34-820b-f85f5e0563e1" (UID: "62b9483c-0a8e-4e34-820b-f85f5e0563e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:20.045970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.018189 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:20.046666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.018220 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:20.112415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.112366 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b9483c-0a8e-4e34-820b-f85f5e0563e1-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.229043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.222663 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:20.229043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.223004 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:15:20.259900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.252118 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:20.417551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.417498 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b9483c-0a8e-4e34-820b-f85f5e0563e1-kube-api-access-qslrd" pod "62b9483c-0a8e-4e34-820b-f85f5e0563e1" (UID: "62b9483c-0a8e-4e34-820b-f85f5e0563e1"). InnerVolumeSpecName "kube-api-access-qslrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:20.425090 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:20.425066 4582 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.053s" Oct 30 18:15:20.425365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.425348 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb"] Oct 30 18:15:20.425967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.425951 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:15:20.440228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.440065 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:15:20.447455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.447115 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:20.470766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.470733 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs"] Oct 30 18:15:20.471886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.471863 4582 cpu_manager.go:414] "containerMap: removing container" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.472033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472017 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.472154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472143 4582 cpu_manager.go:414] "containerMap: removing container" podUID="481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" containerName="revoke-keys" Oct 30 18:15:20.472232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472222 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" containerName="revoke-keys" Oct 30 18:15:20.472369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472358 4582 cpu_manager.go:414] "containerMap: removing container" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.472451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472438 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.472562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472551 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" containerName="revoke-keys" Oct 30 18:15:20.472647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472635 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" containerName="revoke-keys" Oct 30 18:15:20.472730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472719 4582 cpu_manager.go:414] "containerMap: removing container" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.472801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.472791 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.473085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.473069 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.473210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.473198 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" containerName="revoke-keys" Oct 30 18:15:20.473285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.473275 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.473483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.473471 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" containerName="cleanup" Oct 30 18:15:20.473577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.473566 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" containerName="revoke-keys" Oct 30 18:15:20.477348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.477165 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:20.519237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.519086 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qslrd\" (UniqueName: \"kubernetes.io/projected/62b9483c-0a8e-4e34-820b-f85f5e0563e1-kube-api-access-qslrd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.547087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.547054 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:15:20.547323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.547304 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs"] Oct 30 18:15:20.547485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.547467 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:15:20.547645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.547629 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:15:20.547853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.547837 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.552716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.552043 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:15:20.561633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.561616 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:20.573890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.573868 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8"] Oct 30 18:15:20.580427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.580409 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.582547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.582527 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8"] Oct 30 18:15:20.606438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.606417 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:20.672000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.671901 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" Oct 30 18:15:20.672129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.672090 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw" event={"ID":"b43a9bd6-dd57-4327-830c-64ed31ea8da4","Type":"ContainerDied","Data":"b683f1b529ca8274a80524e7f50cc63890f69200222ec22d2393fbdf466be285"} Oct 30 18:15:20.672185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.672169 4582 scope.go:122] "RemoveContainer" containerID="6b3a0c43fd3b019a0c919ec8d19c84d2772ef9752e58c2ff5439f928212534ed" Oct 30 18:15:20.678471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.678438 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" event={"ID":"62b9483c-0a8e-4e34-820b-f85f5e0563e1","Type":"ContainerDied","Data":"4f64f856ef5fc6daf88cda8b6dbbc13847479459c2703ebd10e4fc7313168234"} Oct 30 18:15:20.678594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.678483 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg" Oct 30 18:15:20.701799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.701777 4582 scope.go:122] "RemoveContainer" containerID="3f701874702b7186ea3bc2c9ff5eb02d8f886096079c830f414d146ab27eb31a" Oct 30 18:15:20.705323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.705300 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:20.709371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.709345 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:20.722199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722169 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b43a9bd6-dd57-4327-830c-64ed31ea8da4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43a9bd6-dd57-4327-830c-64ed31ea8da4-maas-api-service-ca\") pod \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " Oct 30 18:15:20.722344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722231 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43a9bd6-dd57-4327-830c-64ed31ea8da4-kube-api-access-glqvf\" (UniqueName: \"kubernetes.io/projected/b43a9bd6-dd57-4327-830c-64ed31ea8da4-kube-api-access-glqvf\") pod \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\" (UID: \"b43a9bd6-dd57-4327-830c-64ed31ea8da4\") " Oct 30 18:15:20.722506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722482 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43a9bd6-dd57-4327-830c-64ed31ea8da4-maas-api-service-ca" pod "b43a9bd6-dd57-4327-830c-64ed31ea8da4" (UID: "b43a9bd6-dd57-4327-830c-64ed31ea8da4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:20.722506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722490 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.722606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722539 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.722679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722658 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ccdd\" (UniqueName: \"kubernetes.io/projected/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-kube-api-access-4ccdd\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.722735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722720 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.722955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722938 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l998s\" (UniqueName: \"kubernetes.io/projected/3cf8c547-3c30-4a28-b85d-0496605b81f7-kube-api-access-l998s\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.722996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.722975 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.723059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.723049 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43a9bd6-dd57-4327-830c-64ed31ea8da4-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.724252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.724234 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43a9bd6-dd57-4327-830c-64ed31ea8da4-kube-api-access-glqvf" pod "b43a9bd6-dd57-4327-830c-64ed31ea8da4" (UID: "b43a9bd6-dd57-4327-830c-64ed31ea8da4"). InnerVolumeSpecName "kube-api-access-glqvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:20.823454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.823375 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-kube-api-access-wpq6w\" (UniqueName: \"kubernetes.io/projected/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-kube-api-access-wpq6w\") pod \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " Oct 30 18:15:20.823454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.823444 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-maas-api-service-ca\") pod \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\" (UID: \"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d\") " Oct 30 18:15:20.823665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.823529 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8704aa-cb73-4c59-a639-4582df054e7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8704aa-cb73-4c59-a639-4582df054e7c-maas-api-service-ca\") pod \"9a8704aa-cb73-4c59-a639-4582df054e7c\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " Oct 30 18:15:20.823731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.823710 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8704aa-cb73-4c59-a639-4582df054e7c-kube-api-access-9jvfs\" (UniqueName: \"kubernetes.io/projected/9a8704aa-cb73-4c59-a639-4582df054e7c-kube-api-access-9jvfs\") pod \"9a8704aa-cb73-4c59-a639-4582df054e7c\" (UID: \"9a8704aa-cb73-4c59-a639-4582df054e7c\") " Oct 30 18:15:20.823981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.823954 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-maas-api-service-ca" pod "7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" (UID: "7a3f253e-6c97-418c-8cb5-6e95c25e3c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:20.824077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824025 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8704aa-cb73-4c59-a639-4582df054e7c-maas-api-service-ca" pod "9a8704aa-cb73-4c59-a639-4582df054e7c" (UID: "9a8704aa-cb73-4c59-a639-4582df054e7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:20.824145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824119 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l998s\" (UniqueName: \"kubernetes.io/projected/3cf8c547-3c30-4a28-b85d-0496605b81f7-kube-api-access-l998s\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.824201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824173 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.824280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824215 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.824280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824239 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.824438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824403 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ccdd\" (UniqueName: \"kubernetes.io/projected/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-kube-api-access-4ccdd\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.824534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824515 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.825866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824884 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqvf\" (UniqueName: \"kubernetes.io/projected/b43a9bd6-dd57-4327-830c-64ed31ea8da4-kube-api-access-glqvf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.825866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824915 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.825866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.824930 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8704aa-cb73-4c59-a639-4582df054e7c-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.825866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.825291 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.826230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.825921 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.827975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.827941 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-kube-api-access-wpq6w" pod "7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" (UID: "7a3f253e-6c97-418c-8cb5-6e95c25e3c1d"). InnerVolumeSpecName "kube-api-access-wpq6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:20.828111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.828040 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8704aa-cb73-4c59-a639-4582df054e7c-kube-api-access-9jvfs" pod "9a8704aa-cb73-4c59-a639-4582df054e7c" (UID: "9a8704aa-cb73-4c59-a639-4582df054e7c"). InnerVolumeSpecName "kube-api-access-9jvfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:20.828385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.828360 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.829536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.829513 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.840279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.840247 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l998s\" (UniqueName: \"kubernetes.io/projected/3cf8c547-3c30-4a28-b85d-0496605b81f7-kube-api-access-l998s\") pod \"maas-api-revoke-sub-7fa2a1ed-t56h8\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.840471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.840448 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ccdd\" (UniqueName: \"kubernetes.io/projected/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-kube-api-access-4ccdd\") pod \"maas-api-revoke-sub-7263f035-h9nvs\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.881975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.881946 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:20.896962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.896933 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg"] Oct 30 18:15:20.900368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.900343 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889735-zxrqg"] Oct 30 18:15:20.920801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.920597 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:20.930186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.930119 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpq6w\" (UniqueName: \"kubernetes.io/projected/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d-kube-api-access-wpq6w\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.930186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:20.930153 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvfs\" (UniqueName: \"kubernetes.io/projected/9a8704aa-cb73-4c59-a639-4582df054e7c-kube-api-access-9jvfs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.022249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.022219 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw"] Oct 30 18:15:21.028336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.028313 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889735-wh2fw"] Oct 30 18:15:21.194010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.193985 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs"] Oct 30 18:15:21.196648 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:21.196616 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502a4996_fc3c_48a6_a77f_b660e8fc9cb9.slice/crio-c4cdd98c5ca0be9ab9339f5d6d0d4216d820fdaa42ae2d4b9fe9dd870c5088ba WatchSource:0}: Error finding container c4cdd98c5ca0be9ab9339f5d6d0d4216d820fdaa42ae2d4b9fe9dd870c5088ba: Status 404 returned error can't find the container with id c4cdd98c5ca0be9ab9339f5d6d0d4216d820fdaa42ae2d4b9fe9dd870c5088ba Oct 30 18:15:21.241878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.241857 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8"] Oct 30 18:15:21.244349 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:21.244325 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf8c547_3c30_4a28_b85d_0496605b81f7.slice/crio-25c7232740630ae0a5ecb79281d47454551ee6df5632f1d949b406cc4966ae4f WatchSource:0}: Error finding container 25c7232740630ae0a5ecb79281d47454551ee6df5632f1d949b406cc4966ae4f: Status 404 returned error can't find the container with id 25c7232740630ae0a5ecb79281d47454551ee6df5632f1d949b406cc4966ae4f Oct 30 18:15:21.301079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.301050 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b9483c-0a8e-4e34-820b-f85f5e0563e1" path="/var/lib/kubelet/pods/62b9483c-0a8e-4e34-820b-f85f5e0563e1/volumes" Oct 30 18:15:21.301579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.301558 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" path="/var/lib/kubelet/pods/b43a9bd6-dd57-4327-830c-64ed31ea8da4/volumes" Oct 30 18:15:21.352623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.352593 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:21.689245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.689210 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" event={"ID":"502a4996-fc3c-48a6-a77f-b660e8fc9cb9","Type":"ContainerStarted","Data":"bf8bd2318455a89e6e4727d0cf9bc422a945626f59b6fb827a56aacf8f4f0709"} Oct 30 18:15:21.689678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.689273 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" event={"ID":"502a4996-fc3c-48a6-a77f-b660e8fc9cb9","Type":"ContainerStarted","Data":"c4cdd98c5ca0be9ab9339f5d6d0d4216d820fdaa42ae2d4b9fe9dd870c5088ba"} Oct 30 18:15:21.690718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.690700 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" Oct 30 18:15:21.690852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.690728 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl" event={"ID":"9a8704aa-cb73-4c59-a639-4582df054e7c","Type":"ContainerDied","Data":"cf4623ce70b8271839ba6f24f0cc7e05d09724f58acfc65a6f0085474e830ddf"} Oct 30 18:15:21.690852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.690785 4582 scope.go:122] "RemoveContainer" containerID="ba01827b4716eb6a7d1b1456c0c4592ead859ee93ecdb77c634bb837587ae571" Oct 30 18:15:21.693599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.693574 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" Oct 30 18:15:21.693599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.693592 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb" event={"ID":"7a3f253e-6c97-418c-8cb5-6e95c25e3c1d","Type":"ContainerDied","Data":"40e8f39f9e87ff20d8b2a8d5f8ea94c0d7e325e1ca7d3e92040e955ad1088e36"} Oct 30 18:15:21.696512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.696492 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" event={"ID":"3cf8c547-3c30-4a28-b85d-0496605b81f7","Type":"ContainerStarted","Data":"b61814cfe4ae5ade5f646ee46308779ad3dd86a64a2de1e0f2b65e9df719cf9b"} Oct 30 18:15:21.696598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.696526 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" event={"ID":"3cf8c547-3c30-4a28-b85d-0496605b81f7","Type":"ContainerStarted","Data":"25c7232740630ae0a5ecb79281d47454551ee6df5632f1d949b406cc4966ae4f"} Oct 30 18:15:21.701578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.701560 4582 scope.go:122] "RemoveContainer" containerID="d1dfe7885b6c31421947e7337538a5a9bb6d98c3caa1b47cc0bd0078d4ebf3e7" Oct 30 18:15:21.723810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.723756 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" podStartSLOduration=4.723740771 podStartE2EDuration="4.723740771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:21.721075206 +0000 UTC m=+1018.545469507" watchObservedRunningTime="2026-10-30 18:15:21.723740771 +0000 UTC m=+1018.548135065" Oct 30 18:15:21.752206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.752183 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl"] Oct 30 18:15:21.757873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.757848 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-nrwzl"] Oct 30 18:15:21.782192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.782168 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb"] Oct 30 18:15:21.786410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.786392 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889735-gn8fb"] Oct 30 18:15:21.822553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:21.822504 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" podStartSLOduration=2.822488328 podStartE2EDuration="2.822488328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:21.816652923 +0000 UTC m=+1018.641047215" watchObservedRunningTime="2026-10-30 18:15:21.822488328 +0000 UTC m=+1018.646882622" Oct 30 18:15:22.705575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.705542 4582 generic.go:356] "Generic (PLEG): container finished" podID="502a4996-fc3c-48a6-a77f-b660e8fc9cb9" containerID="bf8bd2318455a89e6e4727d0cf9bc422a945626f59b6fb827a56aacf8f4f0709" exitCode=0 Oct 30 18:15:22.705981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.705628 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" event={"ID":"502a4996-fc3c-48a6-a77f-b660e8fc9cb9","Type":"ContainerDied","Data":"bf8bd2318455a89e6e4727d0cf9bc422a945626f59b6fb827a56aacf8f4f0709"} Oct 30 18:15:22.709153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.709120 4582 generic.go:356] "Generic (PLEG): container finished" podID="3cf8c547-3c30-4a28-b85d-0496605b81f7" containerID="b61814cfe4ae5ade5f646ee46308779ad3dd86a64a2de1e0f2b65e9df719cf9b" exitCode=0 Oct 30 18:15:22.709286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.709202 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" event={"ID":"3cf8c547-3c30-4a28-b85d-0496605b81f7","Type":"ContainerDied","Data":"b61814cfe4ae5ade5f646ee46308779ad3dd86a64a2de1e0f2b65e9df719cf9b"} Oct 30 18:15:22.845259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.845224 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg"] Oct 30 18:15:22.845958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.845942 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.845962 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.845978 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.845983 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.845992 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:22.846025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846000 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:22.846025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846010 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846014 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846030 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846035 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846050 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846055 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846063 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846067 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846082 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846086 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846183 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846192 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846198 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846207 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846216 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846223 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.846586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846229 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:22.846586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.846235 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" containerName="cleanup" Oct 30 18:15:22.849452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.849435 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.870907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.870881 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg"] Oct 30 18:15:22.956111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956023 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956092 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956193 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956253 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956302 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzdr\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-kube-api-access-rxzdr\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956325 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956447 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956525 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:22.956596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:22.956565 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.044331 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.044307 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:23.058272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058248 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058295 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058330 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058367 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058420 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058478 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058510 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058557 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzdr\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-kube-api-access-rxzdr\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058589 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058769 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058871 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.058912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058882 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.059033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.058980 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.059144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.059090 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.060658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.060637 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.060866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.060847 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.077477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.077457 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.078927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.078902 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzdr\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-kube-api-access-rxzdr\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.168031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.167985 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:23.310308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.310275 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3f253e-6c97-418c-8cb5-6e95c25e3c1d" path="/var/lib/kubelet/pods/7a3f253e-6c97-418c-8cb5-6e95c25e3c1d/volumes" Oct 30 18:15:23.310727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.310712 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8704aa-cb73-4c59-a639-4582df054e7c" path="/var/lib/kubelet/pods/9a8704aa-cb73-4c59-a639-4582df054e7c/volumes" Oct 30 18:15:23.493419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.493391 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg"] Oct 30 18:15:23.493988 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:23.493958 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f6f6d_a76b_42c7_8576_8d233cea91f9.slice/crio-597efca10c495217629e667e7d1ba536ff402bedd3388a1db573ec7114bd04b1 WatchSource:0}: Error finding container 597efca10c495217629e667e7d1ba536ff402bedd3388a1db573ec7114bd04b1: Status 404 returned error can't find the container with id 597efca10c495217629e667e7d1ba536ff402bedd3388a1db573ec7114bd04b1 Oct 30 18:15:23.496347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.496312 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:23.496449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.496391 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:23.496449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.496436 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:23.718513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.718475 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" event={"ID":"a39f6f6d-a76b-42c7-8576-8d233cea91f9","Type":"ContainerStarted","Data":"826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994"} Oct 30 18:15:23.719001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.718537 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" event={"ID":"a39f6f6d-a76b-42c7-8576-8d233cea91f9","Type":"ContainerStarted","Data":"597efca10c495217629e667e7d1ba536ff402bedd3388a1db573ec7114bd04b1"} Oct 30 18:15:23.763912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.761719 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podStartSLOduration=1.76169791 podStartE2EDuration="1.76169791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:23.756143343 +0000 UTC m=+1020.580537637" watchObservedRunningTime="2026-10-30 18:15:23.76169791 +0000 UTC m=+1020.586092204" Oct 30 18:15:23.904376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.904348 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44"] Oct 30 18:15:23.905310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.905288 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:23.905365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.905315 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:23.905517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.905505 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43a9bd6-dd57-4327-830c-64ed31ea8da4" containerName="cleanup" Oct 30 18:15:23.910868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.910848 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.912575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.912554 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:23.916753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.916735 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:23.927859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.927803 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44"] Oct 30 18:15:23.970119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970091 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-openshift-ca\") pod \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " Oct 30 18:15:23.970275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970199 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-kube-api-access-4ccdd\" (UniqueName: \"kubernetes.io/projected/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-kube-api-access-4ccdd\") pod \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " Oct 30 18:15:23.970275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970262 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-openshift-ca\") pod \"3cf8c547-3c30-4a28-b85d-0496605b81f7\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " Oct 30 18:15:23.970382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970283 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf8c547-3c30-4a28-b85d-0496605b81f7-kube-api-access-l998s\" (UniqueName: \"kubernetes.io/projected/3cf8c547-3c30-4a28-b85d-0496605b81f7-kube-api-access-l998s\") pod \"3cf8c547-3c30-4a28-b85d-0496605b81f7\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " Oct 30 18:15:23.970382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970308 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-xks-ca\") pod \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\" (UID: \"502a4996-fc3c-48a6-a77f-b660e8fc9cb9\") " Oct 30 18:15:23.970382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970349 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-xks-ca\") pod \"3cf8c547-3c30-4a28-b85d-0496605b81f7\" (UID: \"3cf8c547-3c30-4a28-b85d-0496605b81f7\") " Oct 30 18:15:23.970526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970477 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-openshift-ca" pod "502a4996-fc3c-48a6-a77f-b660e8fc9cb9" (UID: "502a4996-fc3c-48a6-a77f-b660e8fc9cb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:23.970592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970574 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jmm\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-kube-api-access-m8jmm\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.970645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970619 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.970697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970658 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77961a0e-9b90-46b5-8a93-3047caa3e29d-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.970851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970811 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.970945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970913 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.971010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970958 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.971010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.970969 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-openshift-ca" pod "3cf8c547-3c30-4a28-b85d-0496605b81f7" (UID: "3cf8c547-3c30-4a28-b85d-0496605b81f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:23.971115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.971049 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.971191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.971157 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.971361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.971338 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:23.971499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.971480 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:23.971559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.971507 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:23.972615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.972580 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-xks-ca" pod "3cf8c547-3c30-4a28-b85d-0496605b81f7" (UID: "3cf8c547-3c30-4a28-b85d-0496605b81f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:23.972727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.972645 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-xks-ca" pod "502a4996-fc3c-48a6-a77f-b660e8fc9cb9" (UID: "502a4996-fc3c-48a6-a77f-b660e8fc9cb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:23.972727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.972695 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf8c547-3c30-4a28-b85d-0496605b81f7-kube-api-access-l998s" pod "3cf8c547-3c30-4a28-b85d-0496605b81f7" (UID: "3cf8c547-3c30-4a28-b85d-0496605b81f7"). InnerVolumeSpecName "kube-api-access-l998s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:23.973587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:23.973571 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-kube-api-access-4ccdd" pod "502a4996-fc3c-48a6-a77f-b660e8fc9cb9" (UID: "502a4996-fc3c-48a6-a77f-b660e8fc9cb9"). InnerVolumeSpecName "kube-api-access-4ccdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073112 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jmm\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-kube-api-access-m8jmm\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073178 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073213 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77961a0e-9b90-46b5-8a93-3047caa3e29d-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073288 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073366 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073415 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073442 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073495 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073612 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073704 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ccdd\" (UniqueName: \"kubernetes.io/projected/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-kube-api-access-4ccdd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073724 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l998s\" (UniqueName: \"kubernetes.io/projected/3cf8c547-3c30-4a28-b85d-0496605b81f7-kube-api-access-l998s\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073740 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502a4996-fc3c-48a6-a77f-b660e8fc9cb9-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.073755 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf8c547-3c30-4a28-b85d-0496605b81f7-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:24.074801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.074621 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.076857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.076291 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77961a0e-9b90-46b5-8a93-3047caa3e29d-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.076857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.076535 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.077642 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.077340 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.077642 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.077599 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.080091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.080066 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.080357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.080336 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.092168 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.092147 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.092634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.092616 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jmm\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-kube-api-access-m8jmm\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.172893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.171386 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:24.173289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.173151 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:15:24.173289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.173220 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 18:15:24.227357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.227335 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:24.572218 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:24.572172 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77961a0e_9b90_46b5_8a93_3047caa3e29d.slice/crio-81c1b0845fb75f5ea19466bebaf35a22695fe10a92483253269aa6b28680f990 WatchSource:0}: Error finding container 81c1b0845fb75f5ea19466bebaf35a22695fe10a92483253269aa6b28680f990: Status 404 returned error can't find the container with id 81c1b0845fb75f5ea19466bebaf35a22695fe10a92483253269aa6b28680f990 Oct 30 18:15:24.572892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.572859 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44"] Oct 30 18:15:24.575381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.575345 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:24.575479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.575432 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:24.575479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.575465 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:24.730112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.730087 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" Oct 30 18:15:24.730568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.730086 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8" event={"ID":"3cf8c547-3c30-4a28-b85d-0496605b81f7","Type":"ContainerDied","Data":"25c7232740630ae0a5ecb79281d47454551ee6df5632f1d949b406cc4966ae4f"} Oct 30 18:15:24.730568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.730189 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c7232740630ae0a5ecb79281d47454551ee6df5632f1d949b406cc4966ae4f" Oct 30 18:15:24.732488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.732470 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" Oct 30 18:15:24.732645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.732519 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs" event={"ID":"502a4996-fc3c-48a6-a77f-b660e8fc9cb9","Type":"ContainerDied","Data":"c4cdd98c5ca0be9ab9339f5d6d0d4216d820fdaa42ae2d4b9fe9dd870c5088ba"} Oct 30 18:15:24.732645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.732576 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4cdd98c5ca0be9ab9339f5d6d0d4216d820fdaa42ae2d4b9fe9dd870c5088ba" Oct 30 18:15:24.741404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.741166 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" event={"ID":"77961a0e-9b90-46b5-8a93-3047caa3e29d","Type":"ContainerStarted","Data":"da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836"} Oct 30 18:15:24.741404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.741215 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" event={"ID":"77961a0e-9b90-46b5-8a93-3047caa3e29d","Type":"ContainerStarted","Data":"81c1b0845fb75f5ea19466bebaf35a22695fe10a92483253269aa6b28680f990"} Oct 30 18:15:24.792030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.791968 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podStartSLOduration=1.791953603 podStartE2EDuration="1.791953603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:24.78639681 +0000 UTC m=+1021.610791103" watchObservedRunningTime="2026-10-30 18:15:24.791953603 +0000 UTC m=+1021.616347896" Oct 30 18:15:24.819403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:24.819309 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:25.179920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:25.172269 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:15:25.179920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:25.172330 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 18:15:25.228093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:25.228054 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:25.229400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:25.229372 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:15:25.229501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:25.229419 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 18:15:26.170850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.170263 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.170850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.170325 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 18:15:26.228517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.228483 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.228684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.228536 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 18:15:26.583897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.577339 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:26.998855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.997933 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t"] Oct 30 18:15:26.998855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.998761 4582 cpu_manager.go:414] "containerMap: removing container" podUID="502a4996-fc3c-48a6-a77f-b660e8fc9cb9" containerName="revoke-keys" Oct 30 18:15:26.998855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.998784 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502a4996-fc3c-48a6-a77f-b660e8fc9cb9" containerName="revoke-keys" Oct 30 18:15:26.998855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.998860 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3cf8c547-3c30-4a28-b85d-0496605b81f7" containerName="revoke-keys" Oct 30 18:15:26.999338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.998870 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf8c547-3c30-4a28-b85d-0496605b81f7" containerName="revoke-keys" Oct 30 18:15:26.999338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.999065 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf8c547-3c30-4a28-b85d-0496605b81f7" containerName="revoke-keys" Oct 30 18:15:26.999338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:26.999091 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="502a4996-fc3c-48a6-a77f-b660e8fc9cb9" containerName="revoke-keys" Oct 30 18:15:27.006130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.006108 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023151 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t"] Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023267 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023340 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023537 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023607 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023693 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wnk\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-kube-api-access-25wnk\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023793 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023910 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.023957 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.024415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.024016 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126164 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126233 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wnk\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-kube-api-access-25wnk\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126299 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126346 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126388 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126445 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126507 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126561 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126594 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126658 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126696 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126758 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.126845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.126839 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.127155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.127132 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.129096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.129072 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.130221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.130202 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.182843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.169389 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.182843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.169448 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 18:15:27.186752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.186685 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wnk\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-kube-api-access-25wnk\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.199424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.199322 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.228414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.228159 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.228414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.228215 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 18:15:27.331391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.331314 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:27.731676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.731646 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t"] Oct 30 18:15:27.733073 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:27.733037 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a0df4c_9172_4457_95c8_e47ed3da8cf3.slice/crio-c3a2ca4f6a67ac183d013c170e319f0f52ac60f137949933ed4fd35fdf6d124f WatchSource:0}: Error finding container c3a2ca4f6a67ac183d013c170e319f0f52ac60f137949933ed4fd35fdf6d124f: Status 404 returned error can't find the container with id c3a2ca4f6a67ac183d013c170e319f0f52ac60f137949933ed4fd35fdf6d124f Oct 30 18:15:27.735678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.735647 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:27.735771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.735712 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:27.735771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.735741 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:27.816354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:27.816324 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" event={"ID":"10a0df4c-9172-4457-95c8-e47ed3da8cf3","Type":"ContainerStarted","Data":"c3a2ca4f6a67ac183d013c170e319f0f52ac60f137949933ed4fd35fdf6d124f"} Oct 30 18:15:28.172846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:28.172797 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.173114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:28.173084 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 18:15:28.228186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:28.228148 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.228669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:28.228205 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 18:15:28.476374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:28.476290 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:28.834042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:28.833952 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" event={"ID":"10a0df4c-9172-4457-95c8-e47ed3da8cf3","Type":"ContainerStarted","Data":"06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd"} Oct 30 18:15:28.915845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:28.912394 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podStartSLOduration=2.9123719279999998 podStartE2EDuration="2.912371928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:28.894105167 +0000 UTC m=+1025.718499528" watchObservedRunningTime="2026-10-30 18:15:28.912371928 +0000 UTC m=+1025.736766220" Oct 30 18:15:29.082658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.073085 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw"] Oct 30 18:15:29.114922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.100098 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.117227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.117201 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw"] Oct 30 18:15:29.123848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.123158 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:15:29.124029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.123903 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:29.124029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.124023 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:15:29.124162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.124094 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:15:29.223880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.223518 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.223880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.223573 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.223880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.223654 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6p4\" (UniqueName: \"kubernetes.io/projected/23e65248-8fee-4bd4-99e4-4374a6c49169-kube-api-access-jn6p4\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.226133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.226100 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.226283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.226153 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 18:15:29.228242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.228210 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.228538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.228253 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 18:15:29.325054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.325016 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.325054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.325059 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.325327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.325123 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6p4\" (UniqueName: \"kubernetes.io/projected/23e65248-8fee-4bd4-99e4-4374a6c49169-kube-api-access-jn6p4\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.326032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.326001 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.328996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.328949 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.333644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.333372 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:29.334546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.334528 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.334619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.334568 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:29.410845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.405537 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6p4\" (UniqueName: \"kubernetes.io/projected/23e65248-8fee-4bd4-99e4-4374a6c49169-kube-api-access-jn6p4\") pod \"maas-api-revoke-sub-ada9444e-8mxlw\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.455882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.455848 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:29.887857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:29.887803 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw"] Oct 30 18:15:29.909340 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:29.909298 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e65248_8fee_4bd4_99e4_4374a6c49169.slice/crio-04776723b9c5f37f90dbfdde127dfba54adc3ae3f195abc457cc1db3a96365d6 WatchSource:0}: Error finding container 04776723b9c5f37f90dbfdde127dfba54adc3ae3f195abc457cc1db3a96365d6: Status 404 returned error can't find the container with id 04776723b9c5f37f90dbfdde127dfba54adc3ae3f195abc457cc1db3a96365d6 Oct 30 18:15:30.175893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.175279 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.175893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.175342 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 18:15:30.227859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.227798 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.228061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.227874 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 18:15:30.285497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.285462 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:30.331851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.331802 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.332020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.331872 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:30.880841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.880457 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" event={"ID":"23e65248-8fee-4bd4-99e4-4374a6c49169","Type":"ContainerStarted","Data":"4f6789448755db741c9ebc3862338a945b866dbf222f0d10488c85cf7b6b66bb"} Oct 30 18:15:30.880841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.880521 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" event={"ID":"23e65248-8fee-4bd4-99e4-4374a6c49169","Type":"ContainerStarted","Data":"04776723b9c5f37f90dbfdde127dfba54adc3ae3f195abc457cc1db3a96365d6"} Oct 30 18:15:30.921748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:30.921676 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" podStartSLOduration=1.921658673 podStartE2EDuration="1.921658673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:30.914897362 +0000 UTC m=+1027.739291653" watchObservedRunningTime="2026-10-30 18:15:30.921658673 +0000 UTC m=+1027.746052965" Oct 30 18:15:31.111730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.111695 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs"] Oct 30 18:15:31.134199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.134121 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs"] Oct 30 18:15:31.134351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.134227 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.190495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.190458 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.190834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.190797 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.191254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.191234 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjnnf\" (UniqueName: \"kubernetes.io/projected/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-kube-api-access-tjnnf\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.228287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.228253 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.228534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.228506 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 18:15:31.298854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.292688 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.322663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.321700 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.323506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.322924 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.327305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.324692 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjnnf\" (UniqueName: \"kubernetes.io/projected/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-kube-api-access-tjnnf\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.340159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.340097 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.340455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.340414 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:31.397839 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.397744 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.450548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.448583 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjnnf\" (UniqueName: \"kubernetes.io/projected/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-kube-api-access-tjnnf\") pod \"maas-api-revoke-sub-d7c60ef7-x27cs\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.491940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.491631 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:31.566166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.561019 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:31.566166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.564355 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:31.566166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.565289 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:15:31.887849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.887710 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs"] Oct 30 18:15:31.890275 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:31.890242 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcdd1c2_6ccb_44e6_b568_797a68a9ef31.slice/crio-a55250f36811946ad3fc9e98836ab077ab1a97792fc99ef2e90ba9c21eafd9a9 WatchSource:0}: Error finding container a55250f36811946ad3fc9e98836ab077ab1a97792fc99ef2e90ba9c21eafd9a9: Status 404 returned error can't find the container with id a55250f36811946ad3fc9e98836ab077ab1a97792fc99ef2e90ba9c21eafd9a9 Oct 30 18:15:31.891811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.891764 4582 generic.go:356] "Generic (PLEG): container finished" podID="23e65248-8fee-4bd4-99e4-4374a6c49169" containerID="4f6789448755db741c9ebc3862338a945b866dbf222f0d10488c85cf7b6b66bb" exitCode=0 Oct 30 18:15:31.893204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.893179 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" event={"ID":"23e65248-8fee-4bd4-99e4-4374a6c49169","Type":"ContainerDied","Data":"4f6789448755db741c9ebc3862338a945b866dbf222f0d10488c85cf7b6b66bb"} Oct 30 18:15:31.941502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.941471 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx"] Oct 30 18:15:31.984403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:31.969085 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.021271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.021217 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx"] Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.115686 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.115752 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.115799 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.115843 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.115873 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.115941 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrzh\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-kube-api-access-wvrzh\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.115972 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.116171 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.116232 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1585b9-efd1-4cc2-8643-522e787e6f69-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.218710 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.218787 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1585b9-efd1-4cc2-8643-522e787e6f69-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.218840 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.218889 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.218933 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.218959 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.218984 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.219052 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrzh\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-kube-api-access-wvrzh\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.219082 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.221897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.219527 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.231429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.225246 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.231429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.226085 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.231429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.226327 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.231429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.226480 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1585b9-efd1-4cc2-8643-522e787e6f69-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.282879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.245965 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.282879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.256934 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.282879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.263113 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrzh\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-kube-api-access-wvrzh\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.282879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.263606 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.311988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.311136 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:32.403081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.401784 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:32.403081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.401879 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:32.459444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.457406 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:32.563849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.559374 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:32.963011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.962973 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" event={"ID":"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31","Type":"ContainerStarted","Data":"fc514ec76099307de77f674ab9676aa9a8b45cb6548b521675d068c86ddb5068"} Oct 30 18:15:32.963306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:32.963285 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" event={"ID":"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31","Type":"ContainerStarted","Data":"a55250f36811946ad3fc9e98836ab077ab1a97792fc99ef2e90ba9c21eafd9a9"} Oct 30 18:15:33.025199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.025133 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" podStartSLOduration=2.025110775 podStartE2EDuration="2.025110775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:33.002662524 +0000 UTC m=+1029.827056818" watchObservedRunningTime="2026-10-30 18:15:33.025110775 +0000 UTC m=+1029.849505068" Oct 30 18:15:33.123121 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.122932 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:15:33.123912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.123564 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.248892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.239945 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:33.248892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.240022 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.278844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.274920 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.286330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.285933 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:33.286330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.285995 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.334380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.334345 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:33.334991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.334945 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:33.498806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.498752 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.538899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.537912 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.545799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:33.539854 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.242874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.216935 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:15:34.242874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.217372 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.242874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.218028 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:15:34.303295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.275893 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.303295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.276327 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:15:34.325851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.310932 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:34.325851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.311010 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.387847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.336232 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:34.387847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.336295 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:34.812941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.810076 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:34.969987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:34.969944 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:15:35.093569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.093465 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.094632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.083062 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:15:35.334351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.298989 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.334351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.299295 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:15:35.377327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.355131 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:35.377327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.355985 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:35.389070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.389034 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:35.568020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.560102 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.568020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.401135 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:35.568020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.514076 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:35.568020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.454347 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.605925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.534952 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.605925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.601377 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:35.605925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.601459 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:35.660317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.660288 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:15:35.660560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.660543 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:15:35.660714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.660699 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:35.660850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.660835 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx"] Oct 30 18:15:35.833224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.833177 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.989906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.948428 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:35.992004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:35.983469 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.011952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.009264 4582 generic.go:356] "Generic (PLEG): container finished" podID="7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" containerID="fc514ec76099307de77f674ab9676aa9a8b45cb6548b521675d068c86ddb5068" exitCode=0 Oct 30 18:15:36.049844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.010457 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" event={"ID":"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31","Type":"ContainerDied","Data":"fc514ec76099307de77f674ab9676aa9a8b45cb6548b521675d068c86ddb5068"} Oct 30 18:15:36.111845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.105926 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:36.111845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.105993 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.144846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.136874 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.531447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.213926 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-xks-ca\") pod \"23e65248-8fee-4bd4-99e4-4374a6c49169\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " Oct 30 18:15:36.532100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.532072 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e65248-8fee-4bd4-99e4-4374a6c49169-kube-api-access-jn6p4\" (UniqueName: \"kubernetes.io/projected/23e65248-8fee-4bd4-99e4-4374a6c49169-kube-api-access-jn6p4\") pod \"23e65248-8fee-4bd4-99e4-4374a6c49169\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " Oct 30 18:15:36.532923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.532899 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-openshift-ca\") pod \"23e65248-8fee-4bd4-99e4-4374a6c49169\" (UID: \"23e65248-8fee-4bd4-99e4-4374a6c49169\") " Oct 30 18:15:36.555204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.305996 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-xks-ca" pod "23e65248-8fee-4bd4-99e4-4374a6c49169" (UID: "23e65248-8fee-4bd4-99e4-4374a6c49169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:36.567849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.352854 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:36.568110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.568074 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:36.568253 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.302952 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.592793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.580241 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:36.592793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.580297 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.592793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.214944 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:36.593248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.593218 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:36.593400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.593375 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.618845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.613037 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:36.618845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.615906 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:15:36.635853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.622981 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:15:36.688540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.688500 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-openshift-ca" pod "23e65248-8fee-4bd4-99e4-4374a6c49169" (UID: "23e65248-8fee-4bd4-99e4-4374a6c49169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:36.834900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.796118 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:36.865357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.865323 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e65248-8fee-4bd4-99e4-4374a6c49169-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:36.865582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.844841 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.993684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:36.993642 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:15:37.116085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.115982 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e65248-8fee-4bd4-99e4-4374a6c49169-kube-api-access-jn6p4" pod "23e65248-8fee-4bd4-99e4-4374a6c49169" (UID: "23e65248-8fee-4bd4-99e4-4374a6c49169"). InnerVolumeSpecName "kube-api-access-jn6p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:37.222097 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.212168 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn6p4\" (UniqueName: \"kubernetes.io/projected/23e65248-8fee-4bd4-99e4-4374a6c49169-kube-api-access-jn6p4\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:37.223119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.223078 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.530064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.530029 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:37.532918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.532878 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:37.602135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.545438 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.604226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.602381 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.647100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.647021 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:37.647375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.647344 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.647857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.647814 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.653918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.653893 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:37.735868 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:37.731165 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:15:37.771079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.770914 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.881942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.881896 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.987911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.884227 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" event={"ID":"23e65248-8fee-4bd4-99e4-4374a6c49169","Type":"ContainerDied","Data":"04776723b9c5f37f90dbfdde127dfba54adc3ae3f195abc457cc1db3a96365d6"} Oct 30 18:15:37.987911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.884282 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04776723b9c5f37f90dbfdde127dfba54adc3ae3f195abc457cc1db3a96365d6" Oct 30 18:15:37.987911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:37.884940 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw" Oct 30 18:15:38.062846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.038307 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:38.134844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.075173 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" event={"ID":"2e1585b9-efd1-4cc2-8643-522e787e6f69","Type":"ContainerStarted","Data":"52e886c3088b0300e6d81194fd5eae982637570c17053daace6ba30525c34cf2"} Oct 30 18:15:38.171959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.107928 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:38.218332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.172143 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:38.218332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.175888 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:38.224201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.224160 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:15:38.224419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.224394 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:38.225948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.225833 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:38.231481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.231453 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h"] Oct 30 18:15:38.232580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.232554 4582 cpu_manager.go:414] "containerMap: removing container" podUID="23e65248-8fee-4bd4-99e4-4374a6c49169" containerName="revoke-keys" Oct 30 18:15:38.232708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.232693 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e65248-8fee-4bd4-99e4-4374a6c49169" containerName="revoke-keys" Oct 30 18:15:38.244883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.233103 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e65248-8fee-4bd4-99e4-4374a6c49169" containerName="revoke-keys" Oct 30 18:15:38.285356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.285323 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:15:38.463195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.462745 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:38.463195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.462941 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:15:38.463195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.462975 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h"] Oct 30 18:15:38.463195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.463125 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:38.463195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.463189 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:15:38.525889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.286987 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.527985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.354421 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:15:38.528219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.528191 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.405160 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.540215 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kczlc\" (UniqueName: \"kubernetes.io/projected/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-kube-api-access-kczlc\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.540367 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.541389 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.543458 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.543524 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.422585 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:15:38.560974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.545754 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 18:15:38.649902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.647274 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.649902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.647318 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kczlc\" (UniqueName: \"kubernetes.io/projected/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-kube-api-access-kczlc\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.649902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.647387 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.662656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.662622 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.663020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.662949 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podStartSLOduration=7.66293155 podStartE2EDuration="7.66293155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:38.143648004 +0000 UTC m=+1034.968042296" watchObservedRunningTime="2026-10-30 18:15:38.66293155 +0000 UTC m=+1035.487325844" Oct 30 18:15:38.667384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.667353 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.685534 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:15:38.685417 4582 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e65248_8fee_4bd4_99e4_4374a6c49169.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:15:38.693731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.693650 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kczlc\" (UniqueName: \"kubernetes.io/projected/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-kube-api-access-kczlc\") pod \"maas-api-revoke-sub-e4d1f1c5-bg67h\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.814632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.814598 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:38.974918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.966043 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:38.974918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.967392 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-kube-api-access-tjnnf\" (UniqueName: \"kubernetes.io/projected/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-kube-api-access-tjnnf\") pod \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " Oct 30 18:15:38.974918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.967442 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-xks-ca\") pod \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " Oct 30 18:15:38.974918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.967519 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-openshift-ca\") pod \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\" (UID: \"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31\") " Oct 30 18:15:38.974918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.968364 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-openshift-ca" pod "7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" (UID: "7dcdd1c2-6ccb-44e6-b568-797a68a9ef31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:38.977639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.977595 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-xks-ca" pod "7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" (UID: "7dcdd1c2-6ccb-44e6-b568-797a68a9ef31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:38.978951 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:38.978925 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-kube-api-access-tjnnf" pod "7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" (UID: "7dcdd1c2-6ccb-44e6-b568-797a68a9ef31"). InnerVolumeSpecName "kube-api-access-tjnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:39.082980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.080305 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjnnf\" (UniqueName: \"kubernetes.io/projected/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-kube-api-access-tjnnf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:39.082980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.080343 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:39.082980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.080360 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:39.093466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.093370 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" Oct 30 18:15:39.093466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.093440 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs" event={"ID":"7dcdd1c2-6ccb-44e6-b568-797a68a9ef31","Type":"ContainerDied","Data":"a55250f36811946ad3fc9e98836ab077ab1a97792fc99ef2e90ba9c21eafd9a9"} Oct 30 18:15:39.093688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.093482 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a55250f36811946ad3fc9e98836ab077ab1a97792fc99ef2e90ba9c21eafd9a9" Oct 30 18:15:39.097856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.097143 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" event={"ID":"2e1585b9-efd1-4cc2-8643-522e787e6f69","Type":"ContainerStarted","Data":"4ed0023c60e659eaa281b7ce42613927247ac4dbe91babe39d92cb13c38293e9"} Oct 30 18:15:39.289191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.289123 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h"] Oct 30 18:15:39.289644 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:39.289614 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0d4529_4f7f_4a2d_b3f5_4c6d8f83e8c5.slice/crio-7121b438eee2b9b2dc55fc939e6ce7a6cd6224a5aebef701f1dfaeee1eee6525 WatchSource:0}: Error finding container 7121b438eee2b9b2dc55fc939e6ce7a6cd6224a5aebef701f1dfaeee1eee6525: Status 404 returned error can't find the container with id 7121b438eee2b9b2dc55fc939e6ce7a6cd6224a5aebef701f1dfaeee1eee6525 Oct 30 18:15:39.312795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.312772 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:15:39.312950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.312814 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 18:15:39.884509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.884475 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx"] Oct 30 18:15:39.885138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.885122 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" containerName="revoke-keys" Oct 30 18:15:39.885187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.885140 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" containerName="revoke-keys" Oct 30 18:15:39.885281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.885272 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" containerName="revoke-keys" Oct 30 18:15:39.889054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.889027 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:39.905398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.905369 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx"] Oct 30 18:15:39.992546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.992454 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:39.992546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.992527 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95nv\" (UniqueName: \"kubernetes.io/projected/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-kube-api-access-b95nv\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:39.992755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:39.992568 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.093388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.093356 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.093549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.093430 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95nv\" (UniqueName: \"kubernetes.io/projected/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-kube-api-access-b95nv\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.093549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.093473 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.094035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.094015 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.096031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.096004 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.105027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.105001 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" event={"ID":"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5","Type":"ContainerStarted","Data":"7a430d237920865868fe66859b3fe1278ba7fbcc7a62854125d67e92ae686bd4"} Oct 30 18:15:40.105141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.105058 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" event={"ID":"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5","Type":"ContainerStarted","Data":"7121b438eee2b9b2dc55fc939e6ce7a6cd6224a5aebef701f1dfaeee1eee6525"} Oct 30 18:15:40.109648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.109625 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95nv\" (UniqueName: \"kubernetes.io/projected/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-kube-api-access-b95nv\") pod \"maas-api-revoke-sub-a79aa899-l7mqx\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.167131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.167066 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" podStartSLOduration=4.167044088 podStartE2EDuration="4.167044088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:40.136433628 +0000 UTC m=+1036.960827921" watchObservedRunningTime="2026-10-30 18:15:40.167044088 +0000 UTC m=+1036.991438381" Oct 30 18:15:40.209489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.209459 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:40.233745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.233708 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:40.315444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.313417 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:15:40.315444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.313472 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 18:15:40.332115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.332081 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:40.332269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.332141 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:40.588715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:40.588375 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx"] Oct 30 18:15:40.591647 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:40.591615 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd804c6bc_8b68_4a54_9cf4_4b435e188ab1.slice/crio-8f790f9c790110c8004ca03112fad322f2c0c344c9fe822924c70c90ee3ad38f WatchSource:0}: Error finding container 8f790f9c790110c8004ca03112fad322f2c0c344c9fe822924c70c90ee3ad38f: Status 404 returned error can't find the container with id 8f790f9c790110c8004ca03112fad322f2c0c344c9fe822924c70c90ee3ad38f Oct 30 18:15:41.119879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.119808 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" event={"ID":"d804c6bc-8b68-4a54-9cf4-4b435e188ab1","Type":"ContainerStarted","Data":"69541fa7e178a5fe58ae7b28d75aded0e5603fd4c3fbcf974a87c100f3cc7cd0"} Oct 30 18:15:41.120483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.120250 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" event={"ID":"d804c6bc-8b68-4a54-9cf4-4b435e188ab1","Type":"ContainerStarted","Data":"8f790f9c790110c8004ca03112fad322f2c0c344c9fe822924c70c90ee3ad38f"} Oct 30 18:15:41.121504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.121480 4582 generic.go:356] "Generic (PLEG): container finished" podID="7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" containerID="7a430d237920865868fe66859b3fe1278ba7fbcc7a62854125d67e92ae686bd4" exitCode=0 Oct 30 18:15:41.121638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.121624 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" event={"ID":"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5","Type":"ContainerDied","Data":"7a430d237920865868fe66859b3fe1278ba7fbcc7a62854125d67e92ae686bd4"} Oct 30 18:15:41.178537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.176884 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" podStartSLOduration=2.176863334 podStartE2EDuration="2.176863334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:41.157186606 +0000 UTC m=+1037.981580914" watchObservedRunningTime="2026-10-30 18:15:41.176863334 +0000 UTC m=+1038.001257628" Oct 30 18:15:41.267031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.258442 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:41.267031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.260408 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:41.267031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.262468 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:15:41.313400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.313371 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:15:41.313557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.313421 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 18:15:41.595634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.595562 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:41.697004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.696977 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:41.804772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:41.804734 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:42.129562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.129528 4582 generic.go:356] "Generic (PLEG): container finished" podID="d804c6bc-8b68-4a54-9cf4-4b435e188ab1" containerID="69541fa7e178a5fe58ae7b28d75aded0e5603fd4c3fbcf974a87c100f3cc7cd0" exitCode=0 Oct 30 18:15:42.129982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.129607 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" event={"ID":"d804c6bc-8b68-4a54-9cf4-4b435e188ab1","Type":"ContainerDied","Data":"69541fa7e178a5fe58ae7b28d75aded0e5603fd4c3fbcf974a87c100f3cc7cd0"} Oct 30 18:15:42.313383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.313353 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:15:42.313629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.313603 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 18:15:42.326780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.326759 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:42.425668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.425598 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-kube-api-access-kczlc\" (UniqueName: \"kubernetes.io/projected/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-kube-api-access-kczlc\") pod \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " Oct 30 18:15:42.425793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.425779 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-openshift-ca\") pod \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " Oct 30 18:15:42.425969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.425945 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-xks-ca\") pod \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\" (UID: \"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5\") " Oct 30 18:15:42.426193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.426162 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-openshift-ca" pod "7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" (UID: "7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:42.426553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.426530 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:42.428226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.428199 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-kube-api-access-kczlc" pod "7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" (UID: "7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5"). InnerVolumeSpecName "kube-api-access-kczlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:42.428364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.428341 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-xks-ca" pod "7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" (UID: "7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:42.472440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.472359 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:42.527494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.527455 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:42.527494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.527497 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kczlc\" (UniqueName: \"kubernetes.io/projected/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5-kube-api-access-kczlc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:42.967727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:42.967627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:43.072269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.072222 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:43.137534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.137504 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" Oct 30 18:15:43.137979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.137534 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h" event={"ID":"7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5","Type":"ContainerDied","Data":"7121b438eee2b9b2dc55fc939e6ce7a6cd6224a5aebef701f1dfaeee1eee6525"} Oct 30 18:15:43.137979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.137588 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7121b438eee2b9b2dc55fc939e6ce7a6cd6224a5aebef701f1dfaeee1eee6525" Oct 30 18:15:43.312800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.312767 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:15:43.312987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.312844 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 18:15:43.376250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.376222 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:43.441257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.441226 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-kube-api-access-b95nv\" (UniqueName: \"kubernetes.io/projected/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-kube-api-access-b95nv\") pod \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " Oct 30 18:15:43.441434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.441392 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-xks-ca\") pod \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " Oct 30 18:15:43.441506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.441460 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-openshift-ca\") pod \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\" (UID: \"d804c6bc-8b68-4a54-9cf4-4b435e188ab1\") " Oct 30 18:15:43.447842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.441905 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-openshift-ca" pod "d804c6bc-8b68-4a54-9cf4-4b435e188ab1" (UID: "d804c6bc-8b68-4a54-9cf4-4b435e188ab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:43.447842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.442213 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:43.473263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.473224 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-kube-api-access-b95nv" pod "d804c6bc-8b68-4a54-9cf4-4b435e188ab1" (UID: "d804c6bc-8b68-4a54-9cf4-4b435e188ab1"). InnerVolumeSpecName "kube-api-access-b95nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:43.473402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.473299 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-xks-ca" pod "d804c6bc-8b68-4a54-9cf4-4b435e188ab1" (UID: "d804c6bc-8b68-4a54-9cf4-4b435e188ab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:43.568259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.568123 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:43.568440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.568427 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95nv\" (UniqueName: \"kubernetes.io/projected/d804c6bc-8b68-4a54-9cf4-4b435e188ab1-kube-api-access-b95nv\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:43.915779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.915736 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf"] Oct 30 18:15:43.916349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.916328 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d804c6bc-8b68-4a54-9cf4-4b435e188ab1" containerName="revoke-keys" Oct 30 18:15:43.916349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.916344 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d804c6bc-8b68-4a54-9cf4-4b435e188ab1" containerName="revoke-keys" Oct 30 18:15:43.916487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.916376 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" containerName="revoke-keys" Oct 30 18:15:43.916487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.916382 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" containerName="revoke-keys" Oct 30 18:15:43.916553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.916486 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d804c6bc-8b68-4a54-9cf4-4b435e188ab1" containerName="revoke-keys" Oct 30 18:15:43.916553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.916500 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" containerName="revoke-keys" Oct 30 18:15:43.920111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.920093 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:43.939365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:43.939340 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf"] Oct 30 18:15:44.076797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.076760 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.076797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.076795 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.077029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.076847 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpj4\" (UniqueName: \"kubernetes.io/projected/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-kube-api-access-6xpj4\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.169715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.169629 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" Oct 30 18:15:44.169715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.169647 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx" event={"ID":"d804c6bc-8b68-4a54-9cf4-4b435e188ab1","Type":"ContainerDied","Data":"8f790f9c790110c8004ca03112fad322f2c0c344c9fe822924c70c90ee3ad38f"} Oct 30 18:15:44.169715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.169697 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f790f9c790110c8004ca03112fad322f2c0c344c9fe822924c70c90ee3ad38f" Oct 30 18:15:44.178815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.178787 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.178957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.178860 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.178957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.178889 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpj4\" (UniqueName: \"kubernetes.io/projected/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-kube-api-access-6xpj4\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.179595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.179571 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.181180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.181156 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.202815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.202791 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpj4\" (UniqueName: \"kubernetes.io/projected/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-kube-api-access-6xpj4\") pod \"maas-api-revoke-sub-52f6db4d-lgtlf\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.234314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.234291 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:44.320441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.316677 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:15:44.320441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.316729 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 18:15:44.559166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.559143 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf"] Oct 30 18:15:44.560201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.560169 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:44.561395 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:44.561359 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e92d9aa_b84a_461f_882d_b8fe80bc16e0.slice/crio-660a920ae8c69913c684fb9fc94abe17ac093c3741c0c956e007b6823b27ced7 WatchSource:0}: Error finding container 660a920ae8c69913c684fb9fc94abe17ac093c3741c0c956e007b6823b27ced7: Status 404 returned error can't find the container with id 660a920ae8c69913c684fb9fc94abe17ac093c3741c0c956e007b6823b27ced7 Oct 30 18:15:44.628602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.628568 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx"] Oct 30 18:15:44.636436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.636409 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.655025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.654985 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx"] Oct 30 18:15:44.788375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.788346 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q98c7\" (UniqueName: \"kubernetes.io/projected/8955a515-94c1-42b3-b2c5-bbdfbc268825-kube-api-access-q98c7\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.788553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.788390 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.788553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.788410 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.889911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.889877 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q98c7\" (UniqueName: \"kubernetes.io/projected/8955a515-94c1-42b3-b2c5-bbdfbc268825-kube-api-access-q98c7\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.890064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.889928 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.890064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.889951 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.890575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.890551 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.892280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.892256 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.906111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.906088 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q98c7\" (UniqueName: \"kubernetes.io/projected/8955a515-94c1-42b3-b2c5-bbdfbc268825-kube-api-access-q98c7\") pod \"maas-api-revoke-sub-76531443-8zbpx\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:44.960216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:44.960182 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:45.178773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:45.178726 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" event={"ID":"9e92d9aa-b84a-461f-882d-b8fe80bc16e0","Type":"ContainerStarted","Data":"e943a67c5b696f76e6c3a6fb73bdbffba4c3266166a8ab9b3c93c3cc1e90465b"} Oct 30 18:15:45.179156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:45.178833 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" event={"ID":"9e92d9aa-b84a-461f-882d-b8fe80bc16e0","Type":"ContainerStarted","Data":"660a920ae8c69913c684fb9fc94abe17ac093c3741c0c956e007b6823b27ced7"} Oct 30 18:15:45.214629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:45.214579 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" podStartSLOduration=2.214565944 podStartE2EDuration="2.214565944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:45.210350609 +0000 UTC m=+1042.034744901" watchObservedRunningTime="2026-10-30 18:15:45.214565944 +0000 UTC m=+1042.038960236" Oct 30 18:15:45.265757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:45.265669 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx"] Oct 30 18:15:45.268880 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:45.268853 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8955a515_94c1_42b3_b2c5_bbdfbc268825.slice/crio-4b41ad23e34f83010d255c315adfdfd374d05cd5342f19cfd16fe511f3bd08a5 WatchSource:0}: Error finding container 4b41ad23e34f83010d255c315adfdfd374d05cd5342f19cfd16fe511f3bd08a5: Status 404 returned error can't find the container with id 4b41ad23e34f83010d255c315adfdfd374d05cd5342f19cfd16fe511f3bd08a5 Oct 30 18:15:45.728532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:45.728439 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:45.728958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:45.728931 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:45.730031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:45.730011 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:15:46.189777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.189739 4582 generic.go:356] "Generic (PLEG): container finished" podID="9e92d9aa-b84a-461f-882d-b8fe80bc16e0" containerID="e943a67c5b696f76e6c3a6fb73bdbffba4c3266166a8ab9b3c93c3cc1e90465b" exitCode=0 Oct 30 18:15:46.190184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.189796 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" event={"ID":"9e92d9aa-b84a-461f-882d-b8fe80bc16e0","Type":"ContainerDied","Data":"e943a67c5b696f76e6c3a6fb73bdbffba4c3266166a8ab9b3c93c3cc1e90465b"} Oct 30 18:15:46.192102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.191711 4582 generic.go:356] "Generic (PLEG): container finished" podID="8955a515-94c1-42b3-b2c5-bbdfbc268825" containerID="7f2404d88c4a16e94de9401a14b4de9572f2cd157470be865c4092bb3ed18459" exitCode=0 Oct 30 18:15:46.192102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.191808 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" event={"ID":"8955a515-94c1-42b3-b2c5-bbdfbc268825","Type":"ContainerDied","Data":"7f2404d88c4a16e94de9401a14b4de9572f2cd157470be865c4092bb3ed18459"} Oct 30 18:15:46.192102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.191909 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" event={"ID":"8955a515-94c1-42b3-b2c5-bbdfbc268825","Type":"ContainerStarted","Data":"4b41ad23e34f83010d255c315adfdfd374d05cd5342f19cfd16fe511f3bd08a5"} Oct 30 18:15:46.255752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.255721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:46.957945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.957913 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf"] Oct 30 18:15:46.962336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.962321 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:46.979056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:46.979029 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf"] Oct 30 18:15:47.026238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.026197 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7wh\" (UniqueName: \"kubernetes.io/projected/3e570823-83c1-457b-a606-834516c1fe1c-kube-api-access-lh7wh\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.026366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.026271 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.026366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.026349 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.127552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.127514 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7wh\" (UniqueName: \"kubernetes.io/projected/3e570823-83c1-457b-a606-834516c1fe1c-kube-api-access-lh7wh\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.127724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.127657 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.127724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.127696 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.128485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.128460 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.130075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.130056 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.144035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.144011 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7wh\" (UniqueName: \"kubernetes.io/projected/3e570823-83c1-457b-a606-834516c1fe1c-kube-api-access-lh7wh\") pod \"maas-api-revoke-sub-02440f36-796nf\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.281254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.276371 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:47.398758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.388639 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:47.435865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.434916 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:47.437517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.437496 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-kube-api-access-6xpj4\" (UniqueName: \"kubernetes.io/projected/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-kube-api-access-6xpj4\") pod \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " Oct 30 18:15:47.438426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.437762 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-openshift-ca\") pod \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " Oct 30 18:15:47.438426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.437790 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-xks-ca\") pod \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\" (UID: \"9e92d9aa-b84a-461f-882d-b8fe80bc16e0\") " Oct 30 18:15:47.438861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.438803 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-openshift-ca" pod "9e92d9aa-b84a-461f-882d-b8fe80bc16e0" (UID: "9e92d9aa-b84a-461f-882d-b8fe80bc16e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:47.440087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.440058 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-xks-ca" pod "9e92d9aa-b84a-461f-882d-b8fe80bc16e0" (UID: "9e92d9aa-b84a-461f-882d-b8fe80bc16e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:47.440681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.440663 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-kube-api-access-6xpj4" pod "9e92d9aa-b84a-461f-882d-b8fe80bc16e0" (UID: "9e92d9aa-b84a-461f-882d-b8fe80bc16e0"). InnerVolumeSpecName "kube-api-access-6xpj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:47.538890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.538797 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-openshift-ca\") pod \"8955a515-94c1-42b3-b2c5-bbdfbc268825\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " Oct 30 18:15:47.539047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.538946 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-xks-ca\") pod \"8955a515-94c1-42b3-b2c5-bbdfbc268825\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " Oct 30 18:15:47.539047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.538995 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8955a515-94c1-42b3-b2c5-bbdfbc268825-kube-api-access-q98c7\" (UniqueName: \"kubernetes.io/projected/8955a515-94c1-42b3-b2c5-bbdfbc268825-kube-api-access-q98c7\") pod \"8955a515-94c1-42b3-b2c5-bbdfbc268825\" (UID: \"8955a515-94c1-42b3-b2c5-bbdfbc268825\") " Oct 30 18:15:47.539217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.539192 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-openshift-ca" pod "8955a515-94c1-42b3-b2c5-bbdfbc268825" (UID: "8955a515-94c1-42b3-b2c5-bbdfbc268825"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:47.539653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.539627 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:47.539653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.539653 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:47.539813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.539669 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:47.539813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.539682 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpj4\" (UniqueName: \"kubernetes.io/projected/9e92d9aa-b84a-461f-882d-b8fe80bc16e0-kube-api-access-6xpj4\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:47.541268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.541238 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-xks-ca" pod "8955a515-94c1-42b3-b2c5-bbdfbc268825" (UID: "8955a515-94c1-42b3-b2c5-bbdfbc268825"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:47.541383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.541295 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8955a515-94c1-42b3-b2c5-bbdfbc268825-kube-api-access-q98c7" pod "8955a515-94c1-42b3-b2c5-bbdfbc268825" (UID: "8955a515-94c1-42b3-b2c5-bbdfbc268825"). InnerVolumeSpecName "kube-api-access-q98c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:47.630882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.630857 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf"] Oct 30 18:15:47.633124 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:47.633096 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e570823_83c1_457b_a606_834516c1fe1c.slice/crio-7e5f24520395c57adf51986f1dd059d2833054f5f9845cc4c0ca7db025238cf3 WatchSource:0}: Error finding container 7e5f24520395c57adf51986f1dd059d2833054f5f9845cc4c0ca7db025238cf3: Status 404 returned error can't find the container with id 7e5f24520395c57adf51986f1dd059d2833054f5f9845cc4c0ca7db025238cf3 Oct 30 18:15:47.640775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.640750 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8955a515-94c1-42b3-b2c5-bbdfbc268825-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:47.640865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.640776 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q98c7\" (UniqueName: \"kubernetes.io/projected/8955a515-94c1-42b3-b2c5-bbdfbc268825-kube-api-access-q98c7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:47.939055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:47.939017 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:48.212493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.212458 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" event={"ID":"3e570823-83c1-457b-a606-834516c1fe1c","Type":"ContainerStarted","Data":"4e5f6af1a421f8c6caa507dbad9d348b4c479a7becb99f128c57807194ba3acf"} Oct 30 18:15:48.212625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.212515 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" event={"ID":"3e570823-83c1-457b-a606-834516c1fe1c","Type":"ContainerStarted","Data":"7e5f24520395c57adf51986f1dd059d2833054f5f9845cc4c0ca7db025238cf3"} Oct 30 18:15:48.214057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.214034 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" Oct 30 18:15:48.214057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.214042 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf" event={"ID":"9e92d9aa-b84a-461f-882d-b8fe80bc16e0","Type":"ContainerDied","Data":"660a920ae8c69913c684fb9fc94abe17ac093c3741c0c956e007b6823b27ced7"} Oct 30 18:15:48.214217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.214084 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="660a920ae8c69913c684fb9fc94abe17ac093c3741c0c956e007b6823b27ced7" Oct 30 18:15:48.215509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.215482 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" event={"ID":"8955a515-94c1-42b3-b2c5-bbdfbc268825","Type":"ContainerDied","Data":"4b41ad23e34f83010d255c315adfdfd374d05cd5342f19cfd16fe511f3bd08a5"} Oct 30 18:15:48.215581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.215518 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx" Oct 30 18:15:48.215581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.215531 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b41ad23e34f83010d255c315adfdfd374d05cd5342f19cfd16fe511f3bd08a5" Oct 30 18:15:48.254795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:48.254753 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" podStartSLOduration=2.254741981 podStartE2EDuration="2.254741981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:48.254295187 +0000 UTC m=+1045.078689491" watchObservedRunningTime="2026-10-30 18:15:48.254741981 +0000 UTC m=+1045.079136269" Oct 30 18:15:49.228979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:49.228945 4582 generic.go:356] "Generic (PLEG): container finished" podID="3e570823-83c1-457b-a606-834516c1fe1c" containerID="4e5f6af1a421f8c6caa507dbad9d348b4c479a7becb99f128c57807194ba3acf" exitCode=0 Oct 30 18:15:49.229346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:49.229003 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" event={"ID":"3e570823-83c1-457b-a606-834516c1fe1c","Type":"ContainerDied","Data":"4e5f6af1a421f8c6caa507dbad9d348b4c479a7becb99f128c57807194ba3acf"} Oct 30 18:15:49.643375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:49.643345 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:50.391241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.391208 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:50.479080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.479049 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-xks-ca\") pod \"3e570823-83c1-457b-a606-834516c1fe1c\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " Oct 30 18:15:50.479246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.479099 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e570823-83c1-457b-a606-834516c1fe1c-kube-api-access-lh7wh\" (UniqueName: \"kubernetes.io/projected/3e570823-83c1-457b-a606-834516c1fe1c-kube-api-access-lh7wh\") pod \"3e570823-83c1-457b-a606-834516c1fe1c\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " Oct 30 18:15:50.479321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.479303 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-openshift-ca\") pod \"3e570823-83c1-457b-a606-834516c1fe1c\" (UID: \"3e570823-83c1-457b-a606-834516c1fe1c\") " Oct 30 18:15:50.479665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.479637 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-openshift-ca" pod "3e570823-83c1-457b-a606-834516c1fe1c" (UID: "3e570823-83c1-457b-a606-834516c1fe1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:50.479941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.479926 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:50.481257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.481235 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e570823-83c1-457b-a606-834516c1fe1c-kube-api-access-lh7wh" pod "3e570823-83c1-457b-a606-834516c1fe1c" (UID: "3e570823-83c1-457b-a606-834516c1fe1c"). InnerVolumeSpecName "kube-api-access-lh7wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:50.481305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.481283 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-xks-ca" pod "3e570823-83c1-457b-a606-834516c1fe1c" (UID: "3e570823-83c1-457b-a606-834516c1fe1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:50.580914 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.580884 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e570823-83c1-457b-a606-834516c1fe1c-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:50.581031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:50.580923 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7wh\" (UniqueName: \"kubernetes.io/projected/3e570823-83c1-457b-a606-834516c1fe1c-kube-api-access-lh7wh\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:15:51.245329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:51.245291 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" event={"ID":"3e570823-83c1-457b-a606-834516c1fe1c","Type":"ContainerDied","Data":"7e5f24520395c57adf51986f1dd059d2833054f5f9845cc4c0ca7db025238cf3"} Oct 30 18:15:51.245329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:51.245328 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf" Oct 30 18:15:51.245536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:51.245379 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e5f24520395c57adf51986f1dd059d2833054f5f9845cc4c0ca7db025238cf3" Oct 30 18:15:51.391752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:51.391721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:51.795267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:51.795231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:52.260314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.260282 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:52.363969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.363939 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:52.468665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.468632 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:52.805422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.805388 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44"] Oct 30 18:15:52.806022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806007 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8955a515-94c1-42b3-b2c5-bbdfbc268825" containerName="revoke-keys" Oct 30 18:15:52.806090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806023 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8955a515-94c1-42b3-b2c5-bbdfbc268825" containerName="revoke-keys" Oct 30 18:15:52.806090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806047 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9e92d9aa-b84a-461f-882d-b8fe80bc16e0" containerName="revoke-keys" Oct 30 18:15:52.806090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806052 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e92d9aa-b84a-461f-882d-b8fe80bc16e0" containerName="revoke-keys" Oct 30 18:15:52.806090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806068 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3e570823-83c1-457b-a606-834516c1fe1c" containerName="revoke-keys" Oct 30 18:15:52.806090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806074 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e570823-83c1-457b-a606-834516c1fe1c" containerName="revoke-keys" Oct 30 18:15:52.806316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806166 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e92d9aa-b84a-461f-882d-b8fe80bc16e0" containerName="revoke-keys" Oct 30 18:15:52.806316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806178 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e570823-83c1-457b-a606-834516c1fe1c" containerName="revoke-keys" Oct 30 18:15:52.806316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.806189 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8955a515-94c1-42b3-b2c5-bbdfbc268825" containerName="revoke-keys" Oct 30 18:15:52.810157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.810135 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.833732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.833708 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0c6613ea-openshift-default-dockercfg-twcmm\"" Oct 30 18:15:52.852604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.852570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:52.855787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.855761 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44"] Oct 30 18:15:52.910603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.910565 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57842deb-1907-49f8-a94c-e8a097cf4dbf-istiod-ca-cert\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.910603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.910608 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-data\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.910919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.910627 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-certs\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.910919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.910701 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crl5n\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-kube-api-access-crl5n\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.911031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.910954 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-token\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.911031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.910992 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.911098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.911077 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-podinfo\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.911134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.911120 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-credential-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.911165 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.911153 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-envoy\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:52.938751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:52.938725 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:53.036904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.036864 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57842deb-1907-49f8-a94c-e8a097cf4dbf-istiod-ca-cert\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.038048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.038018 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-data\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.043344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.043318 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-certs\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.046486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.046460 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crl5n\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-kube-api-access-crl5n\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.046749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.046722 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-token\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.046941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.046764 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.046941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.046836 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-podinfo\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.046941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.046890 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-credential-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.047111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.046942 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-envoy\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.050597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.037758 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57842deb-1907-49f8-a94c-e8a097cf4dbf-istiod-ca-cert\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.050597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.048947 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-certs\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.050597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.043226 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-data\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.050597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.049287 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.052606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.052372 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-credential-socket\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.066636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.064918 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7"] Oct 30 18:15:53.079115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.079032 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crl5n\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-kube-api-access-crl5n\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.080344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.080281 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-token\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.098698 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.098662 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7"] Oct 30 18:15:53.098853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.098796 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.115041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.115005 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-envoy\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.123969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.123939 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-podinfo\") pod \"e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.127270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.127208 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:53.193340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.188835 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:53.250461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250431 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.250604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250469 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st24w\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-kube-api-access-st24w\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.250604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250581 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.250737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250692 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.250796 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250754 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb73e19-2022-4912-a893-5f32fca609d9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.250796 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250778 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.250946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250910 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb73e19-2022-4912-a893-5f32fca609d9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.250946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250930 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.251053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.250965 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.352958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.352866 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.352958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.352921 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st24w\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-kube-api-access-st24w\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.353196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353059 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353275 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353403 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353410 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb73e19-2022-4912-a893-5f32fca609d9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353532 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353570 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353572 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb73e19-2022-4912-a893-5f32fca609d9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353617 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.353687 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.354125 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb73e19-2022-4912-a893-5f32fca609d9-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.354377 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.354419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.354413 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.356761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.356737 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb73e19-2022-4912-a893-5f32fca609d9-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.356892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.356812 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.371269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.371235 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st24w\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-kube-api-access-st24w\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.374140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.374117 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.446589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.446562 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44"] Oct 30 18:15:53.448586 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:53.448562 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57842deb_1907_49f8_a94c_e8a097cf4dbf.slice/crio-6ecad3ad951ea4c2669c07be4992bd7bc0f8b6373efffff7e38d1131be2c33d8 WatchSource:0}: Error finding container 6ecad3ad951ea4c2669c07be4992bd7bc0f8b6373efffff7e38d1131be2c33d8: Status 404 returned error can't find the container with id 6ecad3ad951ea4c2669c07be4992bd7bc0f8b6373efffff7e38d1131be2c33d8 Oct 30 18:15:53.449415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.449385 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:53.450966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.450938 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:53.451035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.450998 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:53.451035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.451029 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:53.768710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.768686 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7"] Oct 30 18:15:53.771503 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:15:53.771478 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb73e19_2022_4912_a893_5f32fca609d9.slice/crio-4c1cd4dab37207f99761b7dc58245af23cae049bc8872ee757b4bc4dc0d1a35b WatchSource:0}: Error finding container 4c1cd4dab37207f99761b7dc58245af23cae049bc8872ee757b4bc4dc0d1a35b: Status 404 returned error can't find the container with id 4c1cd4dab37207f99761b7dc58245af23cae049bc8872ee757b4bc4dc0d1a35b Oct 30 18:15:53.774062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.774028 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:53.774156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.774106 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:53.774228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:53.774151 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:15:54.278523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.278480 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" event={"ID":"57842deb-1907-49f8-a94c-e8a097cf4dbf","Type":"ContainerStarted","Data":"49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578"} Oct 30 18:15:54.278736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.278549 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" event={"ID":"57842deb-1907-49f8-a94c-e8a097cf4dbf","Type":"ContainerStarted","Data":"6ecad3ad951ea4c2669c07be4992bd7bc0f8b6373efffff7e38d1131be2c33d8"} Oct 30 18:15:54.280330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.280301 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" event={"ID":"ebb73e19-2022-4912-a893-5f32fca609d9","Type":"ContainerStarted","Data":"194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d"} Oct 30 18:15:54.280431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.280346 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" event={"ID":"ebb73e19-2022-4912-a893-5f32fca609d9","Type":"ContainerStarted","Data":"4c1cd4dab37207f99761b7dc58245af23cae049bc8872ee757b4bc4dc0d1a35b"} Oct 30 18:15:54.323841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.323781 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podStartSLOduration=2.323764464 podStartE2EDuration="2.323764464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:54.32200081 +0000 UTC m=+1051.146395103" watchObservedRunningTime="2026-10-30 18:15:54.323764464 +0000 UTC m=+1051.148158758" Oct 30 18:15:54.367131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.367077 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podStartSLOduration=2.367060564 podStartE2EDuration="2.367060564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:54.361729827 +0000 UTC m=+1051.186124156" watchObservedRunningTime="2026-10-30 18:15:54.367060564 +0000 UTC m=+1051.191454913" Oct 30 18:15:54.452086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.451619 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:15:54.452086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.451755 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:15:54.452086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:54.451802 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:15:55.127693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.127647 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:55.133591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.133549 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:55.167844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.166274 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:55.389972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.381804 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:55.389972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.382042 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:15:55.635844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.635214 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:55.673503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.660058 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:15:55.673503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:55.660121 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:15:56.451021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:56.450957 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:15:56.451669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:56.451635 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:15:56.552673 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:56.552628 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:56.563846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:56.562892 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:15:56.630844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:56.616924 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:56.632402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:56.632358 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:56.894958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:56.894909 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.267063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.266957 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.348859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.335940 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.455652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.455211 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:15:57.455652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.455276 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:15:57.491084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.490993 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:57.578177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.578077 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.579359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.579204 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.580444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.580421 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:15:57.581668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.580899 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:15:57.630791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.630748 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.645017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.634217 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:15:57.709319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.644729 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.709902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.652949 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.710461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.710437 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:15:57.846895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.846521 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:57.906849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.902954 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.906849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:57.903689 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:15:58.079845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.074895 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:58.079845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.075717 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 30 18:15:58.273968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.273915 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.315782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.315749 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:15:58.358532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.332609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:58.358965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.340906 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.384669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.374057 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:15:58.499846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.474201 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:15:58.500475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.500436 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:15:58.612076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.611977 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:15:58.668863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.638884 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.668863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.646984 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:15:58.767846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.720116 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:58.951845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.926610 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.963930 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:58.952468 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:59.206903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.202927 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 18:15:59.273895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.245714 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:59.317840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.317784 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:15:59.369196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.369147 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.455433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.455394 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:15:59.459027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.458939 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:15:59.639848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.639584 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:15:59.665242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.640143 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:15:59.665994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.665947 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.720477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.720383 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:59.954854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:15:59.954788 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.084155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.084066 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-v7b8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:00.084488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.084465 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.094857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.084401 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ghw27" podUID="13292340-3a3f-4a46-90ae-cd5a5670350e" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.25:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.325873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.318225 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:16:00.411844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.386571 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:00.412923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.412878 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.449843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.449090 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.494844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.456203 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:16:00.494844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.456262 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:16:00.520955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.520909 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.648365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.648320 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:16:00.650927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.650896 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:16:00.654190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.654170 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:16:00.707341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.676900 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.813383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.770887 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:00.837841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:00.835025 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:01.178167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.178119 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 18:16:01.264871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.236221 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:01.395845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.372512 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:16:01.416467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.416424 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.465911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.465802 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:16:01.466082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.465919 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:16:01.518455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.518415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:01.550843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.548178 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:16:01.600654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.599692 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:16:01.601902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.601877 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:16:01.603266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.603247 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:16:01.606162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:01.606142 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:16:02.280190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:02.280158 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:02.459843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:02.455156 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:16:02.459843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:02.455215 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:16:02.532374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:02.532305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:02.629260 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:02.629225 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:03.449813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.449780 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:16:03.450307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.449852 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 18:16:03.456988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.456966 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:03.540342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.540305 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw"] Oct 30 18:16:03.549263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.549233 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.551379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.551348 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw"] Oct 30 18:16:03.574987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.574962 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:16:03.575220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.574962 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:16:03.575220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.575008 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:16:03.575220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.575082 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:16:03.621432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.621402 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9vt\" (UniqueName: \"kubernetes.io/projected/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-kube-api-access-dn9vt\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.621623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.621442 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.621674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.621639 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.722537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.722453 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.722705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.722583 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9vt\" (UniqueName: \"kubernetes.io/projected/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-kube-api-access-dn9vt\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.722705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.722611 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.723503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.723474 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.724917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.724894 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.740631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.740609 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9vt\" (UniqueName: \"kubernetes.io/projected/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-kube-api-access-dn9vt\") pod \"maas-api-revoke-sub-db99a5d3-hvvlw\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:03.876097 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:03.876060 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:04.181072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:04.181046 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw"] Oct 30 18:16:04.183718 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:16:04.183693 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e621b0f_72ac_44a2_abe7_a7a72850f6c4.slice/crio-ccd2967f31bc5cea09978226418f89b2472b0546d18460aecaf4913921e8b4ab WatchSource:0}: Error finding container ccd2967f31bc5cea09978226418f89b2472b0546d18460aecaf4913921e8b4ab: Status 404 returned error can't find the container with id ccd2967f31bc5cea09978226418f89b2472b0546d18460aecaf4913921e8b4ab Oct 30 18:16:04.451598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:04.451472 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" event={"ID":"5e621b0f-72ac-44a2-abe7-a7a72850f6c4","Type":"ContainerStarted","Data":"ccd2967f31bc5cea09978226418f89b2472b0546d18460aecaf4913921e8b4ab"} Oct 30 18:16:05.153169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.153130 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:05.463808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.463764 4582 generic.go:356] "Generic (PLEG): container finished" podID="5e621b0f-72ac-44a2-abe7-a7a72850f6c4" containerID="e73d1592209c7cda95b13c9ee31e1d76ee1de47d9548b7108c430da227317a3d" exitCode=0 Oct 30 18:16:05.464382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.463861 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" event={"ID":"5e621b0f-72ac-44a2-abe7-a7a72850f6c4","Type":"ContainerDied","Data":"e73d1592209c7cda95b13c9ee31e1d76ee1de47d9548b7108c430da227317a3d"} Oct 30 18:16:05.490753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.490719 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:05.490997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.490777 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.542995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.542965 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:16:05.543558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.543524 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:16:05.544617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:05.544589 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:16:06.224795 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:16:06.224763 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:16:06.338038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.338006 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx"] Oct 30 18:16:06.342522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.342496 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.360032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.359952 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl"] Oct 30 18:16:06.365906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.365862 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.372117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.372066 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx"] Oct 30 18:16:06.385923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.385360 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl"] Oct 30 18:16:06.463741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.463696 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.463973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.463935 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4802e94-077b-4c9b-a036-44903c74f610-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.464368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464048 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.464368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464198 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.464368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464283 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.464368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464332 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.464577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464388 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.464577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464504 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.464684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464612 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.464750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464728 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4802e94-077b-4c9b-a036-44903c74f610-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.464814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464790 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.464898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464846 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.464951 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464912 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.465006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.464974 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.465064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.465031 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.465118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.465078 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jg7\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-kube-api-access-c4jg7\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.465118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.465096 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfz6j\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-kube-api-access-lfz6j\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.465220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.465140 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.567454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567270 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jg7\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-kube-api-access-c4jg7\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.567651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567488 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfz6j\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-kube-api-access-lfz6j\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.567651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567519 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.567651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567579 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.567856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567688 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4802e94-077b-4c9b-a036-44903c74f610-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.567856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567744 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.567856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567786 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.568031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567858 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.568031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567892 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.568031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567917 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.568031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.567965 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.568031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.568003 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.568283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.568069 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4802e94-077b-4c9b-a036-44903c74f610-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.568283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.568126 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.568283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.568160 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.568283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.568181 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.568283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.568237 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.568635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.568290 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.569232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.569202 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.569495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.569471 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.570033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.569796 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.570325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.570300 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.570763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.570737 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.571280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.571260 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.571412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.571389 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4802e94-077b-4c9b-a036-44903c74f610-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.571700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.571678 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.572110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.572090 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.572805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.572784 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.575767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.575738 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4802e94-077b-4c9b-a036-44903c74f610-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.577241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.577119 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.577241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.577160 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.582108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.582083 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.601863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.601344 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.601863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.601473 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfz6j\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-kube-api-access-lfz6j\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.602118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.602058 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jg7\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-kube-api-access-c4jg7\") pod \"maas-default-gateway-openshift-default-76597d6855-4n5fx\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.602705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.602676 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-9rhcl\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.646193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.646163 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:06.664898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.664875 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:06.695030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.694999 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:06.748092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.748053 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:06.774416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.774382 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-xks-ca\") pod \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " Oct 30 18:16:06.774615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.774571 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-openshift-ca\") pod \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " Oct 30 18:16:06.774708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.774646 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-kube-api-access-dn9vt\" (UniqueName: \"kubernetes.io/projected/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-kube-api-access-dn9vt\") pod \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\" (UID: \"5e621b0f-72ac-44a2-abe7-a7a72850f6c4\") " Oct 30 18:16:06.781744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.781691 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-openshift-ca" pod "5e621b0f-72ac-44a2-abe7-a7a72850f6c4" (UID: "5e621b0f-72ac-44a2-abe7-a7a72850f6c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:06.788305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.786305 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-xks-ca" pod "5e621b0f-72ac-44a2-abe7-a7a72850f6c4" (UID: "5e621b0f-72ac-44a2-abe7-a7a72850f6c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:06.791745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.791723 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-kube-api-access-dn9vt" pod "5e621b0f-72ac-44a2-abe7-a7a72850f6c4" (UID: "5e621b0f-72ac-44a2-abe7-a7a72850f6c4"). InnerVolumeSpecName "kube-api-access-dn9vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.880998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.880951 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.880998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.880976 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9vt\" (UniqueName: \"kubernetes.io/projected/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-kube-api-access-dn9vt\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.880998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:06.880999 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e621b0f-72ac-44a2-abe7-a7a72850f6c4-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:16:07.072942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.072911 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl"] Oct 30 18:16:07.074094 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:16:07.074066 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a665a0_c0d5_4c83_bee2_89706cfa8ad8.slice/crio-00536971aeff31fd5c387e008e6500bc239c3cb2a0f7d3be679f06ab8ffc9945 WatchSource:0}: Error finding container 00536971aeff31fd5c387e008e6500bc239c3cb2a0f7d3be679f06ab8ffc9945: Status 404 returned error can't find the container with id 00536971aeff31fd5c387e008e6500bc239c3cb2a0f7d3be679f06ab8ffc9945 Oct 30 18:16:07.076861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.076799 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:07.076969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.076897 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:07.076969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.076928 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:07.077484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.077461 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx"] Oct 30 18:16:07.093170 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:16:07.093138 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4802e94_077b_4c9b_a036_44903c74f610.slice/crio-aaccff52d6d1c76cb22236aab082886cf89909a4dfebeae08e0ab5c5f0a4da78 WatchSource:0}: Error finding container aaccff52d6d1c76cb22236aab082886cf89909a4dfebeae08e0ab5c5f0a4da78: Status 404 returned error can't find the container with id aaccff52d6d1c76cb22236aab082886cf89909a4dfebeae08e0ab5c5f0a4da78 Oct 30 18:16:07.095396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.095365 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:07.095507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.095443 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:07.095507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.095481 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:07.485488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.485445 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" event={"ID":"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8","Type":"ContainerStarted","Data":"b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1"} Oct 30 18:16:07.485980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.485514 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" event={"ID":"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8","Type":"ContainerStarted","Data":"00536971aeff31fd5c387e008e6500bc239c3cb2a0f7d3be679f06ab8ffc9945"} Oct 30 18:16:07.487479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.487452 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" event={"ID":"5e621b0f-72ac-44a2-abe7-a7a72850f6c4","Type":"ContainerDied","Data":"ccd2967f31bc5cea09978226418f89b2472b0546d18460aecaf4913921e8b4ab"} Oct 30 18:16:07.487479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.487468 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw" Oct 30 18:16:07.487652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.487496 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd2967f31bc5cea09978226418f89b2472b0546d18460aecaf4913921e8b4ab" Oct 30 18:16:07.489502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.489476 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" event={"ID":"f4802e94-077b-4c9b-a036-44903c74f610","Type":"ContainerStarted","Data":"33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90"} Oct 30 18:16:07.489606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.489550 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" event={"ID":"f4802e94-077b-4c9b-a036-44903c74f610","Type":"ContainerStarted","Data":"aaccff52d6d1c76cb22236aab082886cf89909a4dfebeae08e0ab5c5f0a4da78"} Oct 30 18:16:07.523380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.523325 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podStartSLOduration=1.523306137 podStartE2EDuration="1.523306137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:07.521322536 +0000 UTC m=+1064.345716853" watchObservedRunningTime="2026-10-30 18:16:07.523306137 +0000 UTC m=+1064.347700449" Oct 30 18:16:07.558798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.558680 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podStartSLOduration=1.558661566 podStartE2EDuration="1.558661566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:07.556099049 +0000 UTC m=+1064.380493358" watchObservedRunningTime="2026-10-30 18:16:07.558661566 +0000 UTC m=+1064.383055858" Oct 30 18:16:07.676868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.671915 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:07.698052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:07.698009 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:08.463844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:08.462491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:08.683846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:08.682959 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4n5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:16:08.683846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:08.683035 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 18:16:08.698008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:08.697872 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-9rhcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:16:08.698008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:08.697944 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 18:16:09.101483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.101444 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt"] Oct 30 18:16:09.102131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.102117 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5e621b0f-72ac-44a2-abe7-a7a72850f6c4" containerName="revoke-keys" Oct 30 18:16:09.102184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.102132 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e621b0f-72ac-44a2-abe7-a7a72850f6c4" containerName="revoke-keys" Oct 30 18:16:09.102230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.102220 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e621b0f-72ac-44a2-abe7-a7a72850f6c4" containerName="revoke-keys" Oct 30 18:16:09.106386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.106286 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123164 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123197 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123217 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123258 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123283 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123307 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123353 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8q7v\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-kube-api-access-k8q7v\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123462 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.123487 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.129870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.126830 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt"] Oct 30 18:16:09.224592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224558 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224599 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224650 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224668 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224685 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224725 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224750 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224776 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224835 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8q7v\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-kube-api-access-k8q7v\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.224972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224942 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.225070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.224973 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.225781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.225752 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.225936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.225759 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.226024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.226003 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.227374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.227347 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.227499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.227482 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.241641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.241617 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.253892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.242284 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8q7v\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-kube-api-access-k8q7v\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.426537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.426501 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:09.667850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.667185 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4n5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:16:09.667850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.667248 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 18:16:09.699913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.696302 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-9rhcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:16:09.699913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.696359 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 18:16:09.815366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.815333 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt"] Oct 30 18:16:09.817026 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:16:09.816929 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2acf72b_22e4_4fee_91a3_b83fe9a4920e.slice/crio-773ede90a0a74e3a82eee9b6dfa32c04a918ae3870e0711a2d95d030ad5be68e WatchSource:0}: Error finding container 773ede90a0a74e3a82eee9b6dfa32c04a918ae3870e0711a2d95d030ad5be68e: Status 404 returned error can't find the container with id 773ede90a0a74e3a82eee9b6dfa32c04a918ae3870e0711a2d95d030ad5be68e Oct 30 18:16:09.819685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.819655 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:09.819768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.819719 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:09.819768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:09.819748 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:10.316855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.311849 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:10.410022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.407374 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:10.535589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.535550 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" event={"ID":"b2acf72b-22e4-4fee-91a3-b83fe9a4920e","Type":"ContainerStarted","Data":"09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5"} Oct 30 18:16:10.535737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.535615 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" event={"ID":"b2acf72b-22e4-4fee-91a3-b83fe9a4920e","Type":"ContainerStarted","Data":"773ede90a0a74e3a82eee9b6dfa32c04a918ae3870e0711a2d95d030ad5be68e"} Oct 30 18:16:10.598902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.595120 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podStartSLOduration=1.595097919 podStartE2EDuration="1.595097919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.575385284 +0000 UTC m=+1067.399779603" watchObservedRunningTime="2026-10-30 18:16:10.595097919 +0000 UTC m=+1067.419492214" Oct 30 18:16:10.671088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.665906 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4n5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:16:10.671088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.665962 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 18:16:10.696624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.696165 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-9rhcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:16:10.696624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.696228 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 18:16:10.721855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.718756 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:10.825143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.825104 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:10.922350 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:10.922313 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:11.427941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:11.427909 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:11.428812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:11.428791 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.428967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:11.428891 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 18:16:11.666231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:11.666193 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4n5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.666528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:11.666496 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 18:16:11.702133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:11.701932 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-9rhcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.702133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:11.701997 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 18:16:12.427861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.427813 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.428348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.427900 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 18:16:12.637918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.637785 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:12.678003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.677969 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4n5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.735926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.735892 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 18:16:12.736359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.734320 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:12.736359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.732929 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-9rhcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.736359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:12.736300 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 18:16:13.033214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.032662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:13.359475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.359360 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:13.427728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.427691 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.427952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.427758 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 18:16:13.669845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.667116 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4n5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.669845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.667175 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 18:16:13.695631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.695597 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-9rhcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.695793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.695654 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 18:16:13.999481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:13.999383 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl"] Oct 30 18:16:14.022601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.022072 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl"] Oct 30 18:16:14.022601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.022223 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.039457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.037507 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:16:14.086286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.086250 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:16:14.086473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.086331 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:16:14.086473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.086430 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:16:14.137645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.137608 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fr94\" (UniqueName: \"kubernetes.io/projected/11ffac5d-a35b-496d-98b3-c7e41dbf023f-kube-api-access-8fr94\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.137925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.137898 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.138141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.138116 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.240426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.240348 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.240600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.240568 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.240675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.240629 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fr94\" (UniqueName: \"kubernetes.io/projected/11ffac5d-a35b-496d-98b3-c7e41dbf023f-kube-api-access-8fr94\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.241775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.241741 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.278946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.277255 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.282566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.282508 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fr94\" (UniqueName: \"kubernetes.io/projected/11ffac5d-a35b-496d-98b3-c7e41dbf023f-kube-api-access-8fr94\") pod \"maas-api-revoke-sub-de0a51d8-jntpl\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.390409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.390189 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:14.427800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.427769 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.427950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.427881 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 18:16:14.692230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.682320 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4n5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.692230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.682393 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 18:16:14.698353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.698301 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-9rhcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.698500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.698358 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 18:16:14.790746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.789705 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl"] Oct 30 18:16:14.902596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:14.902498 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:15.428847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:15.428194 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.428847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:15.428253 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 18:16:15.621305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:15.621264 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" event={"ID":"11ffac5d-a35b-496d-98b3-c7e41dbf023f","Type":"ContainerStarted","Data":"2f112be764e8f78a78f4c0eb678e87fcdda5f163a064e3906a53a09520d61fa6"} Oct 30 18:16:15.621499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:15.621326 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" event={"ID":"11ffac5d-a35b-496d-98b3-c7e41dbf023f","Type":"ContainerStarted","Data":"4270b61cba19055ca0240f26d9473a21f2789d5954f1040fc6d9fba673842862"} Oct 30 18:16:15.694842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:15.694644 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" podStartSLOduration=2.694624973 podStartE2EDuration="2.694624973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:15.6941737 +0000 UTC m=+1072.518568016" watchObservedRunningTime="2026-10-30 18:16:15.694624973 +0000 UTC m=+1072.519019266" Oct 30 18:16:16.426994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.426952 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.427184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.427024 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 18:16:16.495755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.495720 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:16.496266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.496241 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:16.497264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.497237 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:16:16.539271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.539234 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:16.539575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.539554 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:16.540969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.540946 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:16:16.651324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.651284 4582 generic.go:356] "Generic (PLEG): container finished" podID="11ffac5d-a35b-496d-98b3-c7e41dbf023f" containerID="2f112be764e8f78a78f4c0eb678e87fcdda5f163a064e3906a53a09520d61fa6" exitCode=0 Oct 30 18:16:16.651583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.651544 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" event={"ID":"11ffac5d-a35b-496d-98b3-c7e41dbf023f","Type":"ContainerDied","Data":"2f112be764e8f78a78f4c0eb678e87fcdda5f163a064e3906a53a09520d61fa6"} Oct 30 18:16:16.772351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:16.772324 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:17.866665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.866639 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:17.910210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.910065 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:17.910501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.910481 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:17.911373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.911347 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:16:17.966884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.966785 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ffac5d-a35b-496d-98b3-c7e41dbf023f-kube-api-access-8fr94\" (UniqueName: \"kubernetes.io/projected/11ffac5d-a35b-496d-98b3-c7e41dbf023f-kube-api-access-8fr94\") pod \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " Oct 30 18:16:17.967008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.966919 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-xks-ca\") pod \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " Oct 30 18:16:17.967063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.967001 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-openshift-ca\") pod \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\" (UID: \"11ffac5d-a35b-496d-98b3-c7e41dbf023f\") " Oct 30 18:16:17.967356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.967332 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-openshift-ca" pod "11ffac5d-a35b-496d-98b3-c7e41dbf023f" (UID: "11ffac5d-a35b-496d-98b3-c7e41dbf023f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:17.967955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.967929 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:16:17.968988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.968961 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ffac5d-a35b-496d-98b3-c7e41dbf023f-kube-api-access-8fr94" pod "11ffac5d-a35b-496d-98b3-c7e41dbf023f" (UID: "11ffac5d-a35b-496d-98b3-c7e41dbf023f"). InnerVolumeSpecName "kube-api-access-8fr94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:17.969087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:17.969057 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-xks-ca" pod "11ffac5d-a35b-496d-98b3-c7e41dbf023f" (UID: "11ffac5d-a35b-496d-98b3-c7e41dbf023f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:18.068970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:18.068936 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffac5d-a35b-496d-98b3-c7e41dbf023f-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:16:18.068970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:18.068965 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fr94\" (UniqueName: \"kubernetes.io/projected/11ffac5d-a35b-496d-98b3-c7e41dbf023f-kube-api-access-8fr94\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:16:18.677838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:18.672550 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" Oct 30 18:16:18.677838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:18.672671 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl" event={"ID":"11ffac5d-a35b-496d-98b3-c7e41dbf023f","Type":"ContainerDied","Data":"4270b61cba19055ca0240f26d9473a21f2789d5954f1040fc6d9fba673842862"} Oct 30 18:16:18.677838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:18.672733 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4270b61cba19055ca0240f26d9473a21f2789d5954f1040fc6d9fba673842862" Oct 30 18:16:18.749188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:18.749146 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:20.357220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:20.357183 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:21.522858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:21.522298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:21.880899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:21.878689 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:22.655846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:22.647150 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:22.671444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:22.668906 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:22.671444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:22.669382 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:22.671444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:22.669582 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:16:22.721297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:22.721250 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.133594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.133543 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.451928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.443990 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.529264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.529230 4582 scope.go:122] "RemoveContainer" containerID="11243bba40d582086ccf0449ac6533a4e6df1797f1a788484ad92e2309085434" Oct 30 18:16:23.649417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.649375 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:16:23.673844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.650320 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:16:23.676251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.676217 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.702960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.676902 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.702960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.676966 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.702960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.677108 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:16:23.702960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.700855 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:16:23.702960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.700917 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:16:23.752879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.722040 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:23.753997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.753970 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:16:23.803847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:23.802859 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:24.196966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.196919 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.218068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.197889 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:16:24.715872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.418416 4582 scope.go:122] "RemoveContainer" containerID="9b6933f3f4b8de1cea82b6dbd37bfd8d9dd41e5e37f3db8b7c939c773588e87b" Oct 30 18:16:24.715872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.506516 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:24.715872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.695271 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.715872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.683699 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.715872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.665995 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:16:24.715872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.712632 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.715872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.712981 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:16:24.757969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:24.757926 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.201847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.189784 4582 scope.go:122] "RemoveContainer" containerID="05655494b869893f770ae460feee1ec8eaf539a7454e3db507a7e1d80f87cba4" Oct 30 18:16:25.201847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.201524 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.459034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.449359 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.697848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.660095 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:25.697848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.660181 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.698627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.698583 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.736299 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.736205 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:16:25.768180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.736363 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.769367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:25.764952 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.137719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.137686 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:16:26.204941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.204896 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.420464 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.420374 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:26.420715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.420683 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.466974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.460445 4582 scope.go:122] "RemoveContainer" containerID="53effe18ddb26a172588ce750f5e6b2ea1b75be6ccccfc387b9799829f10addc" Oct 30 18:16:26.466974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.460565 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.466974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.461877 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:16:26.759104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.758980 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:16:26.778161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.773991 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.805896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.774319 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.808653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:26.776354 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:16:27.104812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.087169 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:27.243107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.243063 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:27.617868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.594539 4582 scope.go:122] "RemoveContainer" containerID="5c6613b3ba3a8238aeb388448222da3b01dd96b8884a8f206956ea8ee577eb71" Oct 30 18:16:27.762873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.713332 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:27.793025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.792978 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:27.793668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.793636 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 18:16:27.793903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.793877 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:27.794330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.794304 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:27.849810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.849762 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 18:16:27.868898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.853083 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 18:16:27.868898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.853173 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:16:27.877398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.877371 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:27.877637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:27.877589 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:28.099849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.092962 4582 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:28.230913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.183871 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:28.315866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.306700 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:28.733844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.733346 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:28.800225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.800189 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:28.813475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.813437 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:16:28.843652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.825706 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:16:28.883878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.873028 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 18:16:28.908947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.906050 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 18:16:28.934207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:28.934187 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 18:16:29.022389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:29.022365 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 18:16:29.023627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:29.023606 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 18:16:29.023729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:29.023673 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 18:16:29.494484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:29.494451 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:31.290626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:31.290581 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:32.163788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:32.163752 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:33.167091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:33.167046 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:34.832200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:34.832167 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:36.225147 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:16:36.225112 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:16:36.340886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:36.340849 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:37.090223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.090188 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:37.471200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.471162 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:37.501229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.501191 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2"] Oct 30 18:16:37.502233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.502214 4582 cpu_manager.go:414] "containerMap: removing container" podUID="11ffac5d-a35b-496d-98b3-c7e41dbf023f" containerName="revoke-keys" Oct 30 18:16:37.502281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.502237 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ffac5d-a35b-496d-98b3-c7e41dbf023f" containerName="revoke-keys" Oct 30 18:16:37.502412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.502401 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ffac5d-a35b-496d-98b3-c7e41dbf023f" containerName="revoke-keys" Oct 30 18:16:37.508854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.508809 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.525003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.524974 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2"] Oct 30 18:16:37.528371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.528351 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f15587-openshift-default-dockercfg-7jczm\"" Oct 30 18:16:37.573317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573279 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-data\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.573501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573333 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-credential-socket\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.573501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573407 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istiod-ca-cert\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.573614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573502 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-envoy\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.573668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573641 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-certs\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.573727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573670 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-socket\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.573782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573750 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhvq\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-kube-api-access-wqhvq\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.573912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573889 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-podinfo\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.574015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.573997 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-token\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678196 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-certs\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678249 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-socket\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678331 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhvq\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-kube-api-access-wqhvq\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678394 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-podinfo\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678462 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-token\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678555 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-data\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678603 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-credential-socket\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678632 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istiod-ca-cert\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.678690 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-envoy\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.684851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.684624 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-data\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.694149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.689138 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-socket\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.694149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.689356 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-certs\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.694149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.689552 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-credential-socket\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.694149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.690477 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istiod-ca-cert\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.694149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.693215 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-envoy\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.703641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.703612 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-podinfo\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.718659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.718627 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-token\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.732550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.732445 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhvq\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-kube-api-access-wqhvq\") pod \"e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:37.829477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:37.829443 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:38.163008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.162982 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2"] Oct 30 18:16:38.164571 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:16:38.164541 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65c4fcc_dfae_42df_9760_4c8feecd05a3.slice/crio-dac5b9f1ba9b6b8a06f17ba8fb41bc833aa07ed9f8633b26a188870a087e9246 WatchSource:0}: Error finding container dac5b9f1ba9b6b8a06f17ba8fb41bc833aa07ed9f8633b26a188870a087e9246: Status 404 returned error can't find the container with id dac5b9f1ba9b6b8a06f17ba8fb41bc833aa07ed9f8633b26a188870a087e9246 Oct 30 18:16:38.167174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.167139 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:38.167281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.167220 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:38.167281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.167266 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:38.462447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.462354 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" event={"ID":"d65c4fcc-dfae-42df-9760-4c8feecd05a3","Type":"ContainerStarted","Data":"a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f"} Oct 30 18:16:38.462620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.462457 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" event={"ID":"d65c4fcc-dfae-42df-9760-4c8feecd05a3","Type":"ContainerStarted","Data":"dac5b9f1ba9b6b8a06f17ba8fb41bc833aa07ed9f8633b26a188870a087e9246"} Oct 30 18:16:38.484027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.483997 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:38.511757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.511699 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" podStartSLOduration=1.511682944 podStartE2EDuration="1.511682944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:38.511537225 +0000 UTC m=+1095.335931518" watchObservedRunningTime="2026-10-30 18:16:38.511682944 +0000 UTC m=+1095.336077236" Oct 30 18:16:38.570233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.570202 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj"] Oct 30 18:16:38.584211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.584177 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.598282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.598254 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj"] Oct 30 18:16:38.599886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.599861 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b33a49-openshift-default-dockercfg-hw949\"" Oct 30 18:16:38.626615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.626583 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704218 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istiod-ca-cert\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704293 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-data\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704332 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-certs\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704391 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-podinfo\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704448 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-socket\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704481 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-credential-socket\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704518 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-envoy\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704545 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-token\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.706150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.704736 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fl5c\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-kube-api-access-8fl5c\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.817561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.817472 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-socket\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.817837 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.817794 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-credential-socket\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.818025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.818005 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-envoy\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.818150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.818133 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-token\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.834062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.834027 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fl5c\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-kube-api-access-8fl5c\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.834473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.834451 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istiod-ca-cert\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.834643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.834626 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-data\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.834777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.834762 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-certs\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.834944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.834926 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-podinfo\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.836036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.836003 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istiod-ca-cert\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.836172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.828684 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-credential-socket\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.836313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.828936 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-socket\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.847034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.830679 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:38.847034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.838521 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-data\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.847034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.838718 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-certs\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.847034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.846490 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-podinfo\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.853619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.853567 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-envoy\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.896887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.888053 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fl5c\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-kube-api-access-8fl5c\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.898960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.898931 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-token\") pod \"e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.909293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.909264 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:38.929948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:38.929901 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:39.268087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.268050 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj"] Oct 30 18:16:39.278992 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:16:39.278965 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732c9a0e_d8dc_4b8a_ab65_beae822b07da.slice/crio-85deb478d2be16f77f276d704af80a3ece9f95989e17aa704a01d0dea8a11dae WatchSource:0}: Error finding container 85deb478d2be16f77f276d704af80a3ece9f95989e17aa704a01d0dea8a11dae: Status 404 returned error can't find the container with id 85deb478d2be16f77f276d704af80a3ece9f95989e17aa704a01d0dea8a11dae Oct 30 18:16:39.282527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.282489 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:39.282626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.282581 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:39.282667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.282626 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:39.471440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.471406 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" event={"ID":"732c9a0e-d8dc-4b8a-ab65-beae822b07da","Type":"ContainerStarted","Data":"107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1"} Oct 30 18:16:39.471612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.471471 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" event={"ID":"732c9a0e-d8dc-4b8a-ab65-beae822b07da","Type":"ContainerStarted","Data":"85deb478d2be16f77f276d704af80a3ece9f95989e17aa704a01d0dea8a11dae"} Oct 30 18:16:39.511572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.511522 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" podStartSLOduration=1.511509341 podStartE2EDuration="1.511509341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:39.510505737 +0000 UTC m=+1096.334900028" watchObservedRunningTime="2026-10-30 18:16:39.511509341 +0000 UTC m=+1096.335903634" Oct 30 18:16:39.836365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.836329 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:39.909433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:39.909401 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:40.306068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:40.306033 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:40.479483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:40.479451 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:40.480346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:40.480327 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:16:40.916144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:40.916116 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:40.997216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:40.997187 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:41.097625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:41.097592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:41.487187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:41.487150 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:41.487869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:41.487851 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:16:42.185226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:42.185176 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:42.745799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:42.745772 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:42.861512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:42.861481 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:43.117525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.117421 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:43.229680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.227234 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg"] Oct 30 18:16:43.259109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.259013 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:43.259767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.259745 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg"] Oct 30 18:16:43.259949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.259877 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.274012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.272156 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0c6613ea\"/\"kube-root-ca.crt\"" Oct 30 18:16:43.274012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.272191 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0c6613ea\"/\"openshift-service-ca.crt\"" Oct 30 18:16:43.306353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.306328 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0c6613ea\"/\"e2e-lifecycle-model-0c6613ea-kserve-self-signed-certs\"" Oct 30 18:16:43.306484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.306356 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0c6613ea\"/\"default-dockercfg-rq8vw\"" Oct 30 18:16:43.432891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.432858 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b802f37-fda6-4097-b93a-c5a08834f42c-tls-certs\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.433089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.432988 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-tmp-dir\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.433160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.433119 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vkm\" (UniqueName: \"kubernetes.io/projected/9b802f37-fda6-4097-b93a-c5a08834f42c-kube-api-access-76vkm\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.433239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.433211 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-home\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.433290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.433274 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-dshm\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.433460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.433436 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-model-cache\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.534619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.534581 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-model-cache\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.534816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.534697 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b802f37-fda6-4097-b93a-c5a08834f42c-tls-certs\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.534816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.534753 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-tmp-dir\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.534992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.534886 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vkm\" (UniqueName: \"kubernetes.io/projected/9b802f37-fda6-4097-b93a-c5a08834f42c-kube-api-access-76vkm\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.534992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.534923 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-home\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.534992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.534952 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-dshm\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.535157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.535122 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-model-cache\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.535484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.535457 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-tmp-dir\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.535736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.535714 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-home\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.537788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.537768 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-dshm\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.537953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.537934 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b802f37-fda6-4097-b93a-c5a08834f42c-tls-certs\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.553130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.553099 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vkm\" (UniqueName: \"kubernetes.io/projected/9b802f37-fda6-4097-b93a-c5a08834f42c-kube-api-access-76vkm\") pod \"e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.583913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.583876 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:43.899611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:43.899582 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg"] Oct 30 18:16:43.900388 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:16:43.900355 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b802f37_fda6_4097_b93a_c5a08834f42c.slice/crio-ba82c30da14b29056f95a06d5d3e54af77e7ca7c2e995931234ac5e8e76624c7 WatchSource:0}: Error finding container ba82c30da14b29056f95a06d5d3e54af77e7ca7c2e995931234ac5e8e76624c7: Status 404 returned error can't find the container with id ba82c30da14b29056f95a06d5d3e54af77e7ca7c2e995931234ac5e8e76624c7 Oct 30 18:16:44.358573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:44.358496 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:44.523781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:44.523744 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" event={"ID":"9b802f37-fda6-4097-b93a-c5a08834f42c","Type":"ContainerStarted","Data":"3e3516220afd36753e34a46bb5a454d9db294b97ca7e0f3999e9eff242b9b6b1"} Oct 30 18:16:44.524274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:44.523792 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" event={"ID":"9b802f37-fda6-4097-b93a-c5a08834f42c","Type":"ContainerStarted","Data":"ba82c30da14b29056f95a06d5d3e54af77e7ca7c2e995931234ac5e8e76624c7"} Oct 30 18:16:44.559775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:44.559723 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" podStartSLOduration=1.559709502 podStartE2EDuration="1.559709502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:44.557207896 +0000 UTC m=+1101.381602201" watchObservedRunningTime="2026-10-30 18:16:44.559709502 +0000 UTC m=+1101.384103796" Oct 30 18:16:44.584082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:44.584047 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:46.262590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:46.262553 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:48.428488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:48.428454 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:49.440964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:49.440931 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:49.619139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:49.619101 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:50.031383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:50.031349 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf"] Oct 30 18:16:50.040233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:50.040204 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c00d3eca-s5wvf"] Oct 30 18:16:50.333541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:50.333463 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:50.974517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:50.974480 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:51.101253 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:51.101204 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:51.301828 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:51.301744 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f" path="/var/lib/kubelet/pods/5a2ccc93-6b3d-41af-a0b7-6d0f12175a3f/volumes" Oct 30 18:16:52.283131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:52.283096 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:52.748800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:52.748765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:52.855486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:52.855446 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:53.584795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:53.584765 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:53.597757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:53.597728 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:53.639049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:53.639020 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:16:54.255837 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:54.255775 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:55.787991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:55.787945 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:55.867198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:55.867166 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:55.973075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:55.973040 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:56.008461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.008426 4582 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": read tcp 10.132.0.2:44850->10.132.0.105:15021: read: connection reset by peer" start-of-body= Oct 30 18:16:56.008896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.008860 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": read tcp 10.132.0.2:44850->10.132.0.105:15021: read: connection reset by peer" Oct 30 18:16:56.654402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.654365 4582 generic.go:356] "Generic (PLEG): container finished" podID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerID="49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578" exitCode=0 Oct 30 18:16:56.654402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.654409 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" event={"ID":"57842deb-1907-49f8-a94c-e8a097cf4dbf","Type":"ContainerDied","Data":"49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578"} Oct 30 18:16:56.655018 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.655000 4582 scope.go:122] "RemoveContainer" containerID="49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578" Oct 30 18:16:56.656748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.656718 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:56.656863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.656796 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:56.656863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:56.656848 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:16:57.675568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:57.675530 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:57.678635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:57.678596 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" event={"ID":"57842deb-1907-49f8-a94c-e8a097cf4dbf","Type":"ContainerStarted","Data":"33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5"} Oct 30 18:16:58.127425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:58.127394 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:16:58.127584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:58.127563 4582 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:58.127631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:58.127611 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 18:16:59.128523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:59.128489 4582 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:59.129029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:59.128547 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 18:16:59.497919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:16:59.497833 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:00.127785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:00.127752 4582 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:17:00.128000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:00.127805 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 18:17:01.128462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:01.128427 4582 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:17:01.128980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:01.128479 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 18:17:01.284507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:01.284473 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:02.128099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:02.128065 4582 patch_prober.go:29] interesting pod/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:17:02.128286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:02.128128 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 18:17:02.917505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:02.917472 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:03.131657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:03.131628 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:17:03.132114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:03.132094 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:17:03.132779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:03.132761 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:17:04.546246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:04.546212 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:06.115883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:06.115850 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:06.193166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:06.193131 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:06.249203 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:17:06.249153 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:17:06.296659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:06.296615 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:08.156599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:08.156565 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:09.755715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:09.755679 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:11.031225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:11.031193 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7"] Oct 30 18:17:11.041028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:11.040999 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1888862-8glm7"] Oct 30 18:17:11.301326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:11.301252 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195852d3-531a-4256-8eac-953a84533bc8" path="/var/lib/kubelet/pods/195852d3-531a-4256-8eac-953a84533bc8/volumes" Oct 30 18:17:11.463164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:11.463126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:13.254604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:13.254568 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:13.488294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:13.488232 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:15.039514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:15.039482 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s"] Oct 30 18:17:15.062608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:15.062577 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:15.069216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:15.069188 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1bd91c-jkl2s"] Oct 30 18:17:15.301402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:15.301329 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a07f3a1-972c-4f92-bcf0-1f38632b9f17" path="/var/lib/kubelet/pods/3a07f3a1-972c-4f92-bcf0-1f38632b9f17/volumes" Oct 30 18:17:16.368303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:16.368263 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:16.968637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:16.968595 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:17.075664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.073409 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:17.500883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.494873 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7"] Oct 30 18:17:17.503075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.503049 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7"] Oct 30 18:17:17.503361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.503342 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.514250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.512061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:17.517579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.517544 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8dc43c-openshift-default-dockercfg-pghxq\"" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.566743 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-certs\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.566834 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-socket\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.566891 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-token\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.566928 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-podinfo\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.566973 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-data\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.567168 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k86\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-kube-api-access-d7k86\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.567240 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-envoy\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.567273 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfbc4b9-c8ae-4011-869a-a86642c92644-istiod-ca-cert\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.571892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.567313 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-credential-socket\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.608484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.608450 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.690905 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-certs\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.690979 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-socket\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691036 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-token\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691095 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-podinfo\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691141 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-data\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691348 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k86\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-kube-api-access-d7k86\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691423 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-envoy\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691451 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfbc4b9-c8ae-4011-869a-a86642c92644-istiod-ca-cert\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.691844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691491 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-credential-socket\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.692624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.691998 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-credential-socket\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.692624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.692260 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-certs\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.692624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.692482 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-socket\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.753926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.741684 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-data\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.753926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.745180 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfbc4b9-c8ae-4011-869a-a86642c92644-istiod-ca-cert\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.783845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.761533 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-podinfo\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.783845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.779681 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-envoy\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.783845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.782398 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-token\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.816907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.806845 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k86\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-kube-api-access-d7k86\") pod \"e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:17.822985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:17.822720 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:18.265812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:18.265787 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7"] Oct 30 18:17:18.268088 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:17:18.268058 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cfbc4b9_c8ae_4011_869a_a86642c92644.slice/crio-dcd7f998c1e5f1a902aa88e22303246874bb88a4b90baf0eb3d8228b1f7e1c77 WatchSource:0}: Error finding container dcd7f998c1e5f1a902aa88e22303246874bb88a4b90baf0eb3d8228b1f7e1c77: Status 404 returned error can't find the container with id dcd7f998c1e5f1a902aa88e22303246874bb88a4b90baf0eb3d8228b1f7e1c77 Oct 30 18:17:18.270661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:18.270635 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:18.270744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:18.270696 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:18.270744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:18.270732 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:18.738276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:18.738194 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:18.994148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:18.994054 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" event={"ID":"9cfbc4b9-c8ae-4011-869a-a86642c92644","Type":"ContainerStarted","Data":"baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae"} Oct 30 18:17:18.994148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:18.994119 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" event={"ID":"9cfbc4b9-c8ae-4011-869a-a86642c92644","Type":"ContainerStarted","Data":"dcd7f998c1e5f1a902aa88e22303246874bb88a4b90baf0eb3d8228b1f7e1c77"} Oct 30 18:17:19.055764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:19.055715 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podStartSLOduration=2.055701633 podStartE2EDuration="2.055701633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:19.041566081 +0000 UTC m=+1135.865960374" watchObservedRunningTime="2026-10-30 18:17:19.055701633 +0000 UTC m=+1135.880095924" Oct 30 18:17:19.823895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:19.823857 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:19.831186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:19.831157 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:20.002466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:20.002439 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:20.003408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:20.003386 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:17:20.033645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:20.033603 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s"] Oct 30 18:17:20.038338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:20.038314 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273e86f3-nf26s"] Oct 30 18:17:20.428566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:20.428497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:21.303612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:21.303574 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26455faa-a6b9-4845-9d46-4ba0ff02c383" path="/var/lib/kubelet/pods/26455faa-a6b9-4845-9d46-4ba0ff02c383/volumes" Oct 30 18:17:21.434112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:21.434080 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:21.694981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:21.694949 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:22.019447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:22.019351 4582 generic.go:356] "Generic (PLEG): container finished" podID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerID="a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f" exitCode=0 Oct 30 18:17:22.019447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:22.019432 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" event={"ID":"d65c4fcc-dfae-42df-9760-4c8feecd05a3","Type":"ContainerDied","Data":"a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f"} Oct 30 18:17:22.019992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:22.019971 4582 scope.go:122] "RemoveContainer" containerID="a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f" Oct 30 18:17:22.021383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:22.021352 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:22.021491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:22.021430 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:22.021491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:22.021480 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:22.462774 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:22.462734 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:23.029344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:23.029306 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" event={"ID":"d65c4fcc-dfae-42df-9760-4c8feecd05a3","Type":"ContainerStarted","Data":"80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8"} Oct 30 18:17:23.203232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:23.203195 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:23.303597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:23.303517 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:23.829606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:23.829579 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:17:23.830846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:23.830807 4582 patch_prober.go:29] interesting pod/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:17:23.830977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:23.830877 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:17:24.263221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:24.263049 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:24.830379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:24.830343 4582 patch_prober.go:29] interesting pod/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:17:24.830795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:24.830404 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:17:25.027880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.027848 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt"] Oct 30 18:17:25.037064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.036991 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2da5c0-2n5rt"] Oct 30 18:17:25.310608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.310571 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bc93f7-3ad3-4813-9fd1-ecb8a3045726" path="/var/lib/kubelet/pods/e8bc93f7-3ad3-4813-9fd1-ecb8a3045726/volumes" Oct 30 18:17:25.480087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.480043 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:17:25.842581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.842544 4582 patch_prober.go:29] interesting pod/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:17:25.843195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.843164 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:17:25.853912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.853878 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8"] Oct 30 18:17:25.859203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.859177 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:25.906594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.906567 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:17:25.906781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.906768 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:25.907074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.907056 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:17:25.907274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.907259 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:17:25.927240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.927211 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8"] Oct 30 18:17:25.973847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.958399 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s7fn\" (UniqueName: \"kubernetes.io/projected/8741f0df-f42e-4dd6-952e-a67ca1c68d66-kube-api-access-2s7fn\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:25.973847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.972971 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:25.973847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.973166 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:25.997316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:25.997283 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:26.084174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.080928 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:26.084174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.081232 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s7fn\" (UniqueName: \"kubernetes.io/projected/8741f0df-f42e-4dd6-952e-a67ca1c68d66-kube-api-access-2s7fn\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:26.084174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.081314 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:26.084174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.083397 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:26.093838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.093744 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:26.103978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.103949 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s7fn\" (UniqueName: \"kubernetes.io/projected/8741f0df-f42e-4dd6-952e-a67ca1c68d66-kube-api-access-2s7fn\") pod \"maas-api-revoke-sub-69e3303e-2krj8\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:26.229993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.229966 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:26.570992 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:17:26.570960 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8741f0df_f42e_4dd6_952e_a67ca1c68d66.slice/crio-9956890b049dd66d7ffc6c323b9afd324d513dec7e75f73120ed631d11c2744f WatchSource:0}: Error finding container 9956890b049dd66d7ffc6c323b9afd324d513dec7e75f73120ed631d11c2744f: Status 404 returned error can't find the container with id 9956890b049dd66d7ffc6c323b9afd324d513dec7e75f73120ed631d11c2744f Oct 30 18:17:26.572895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.572872 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8"] Oct 30 18:17:26.830223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.830144 4582 patch_prober.go:29] interesting pod/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:17:26.830223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:26.830194 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:17:27.099458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:27.099371 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" event={"ID":"8741f0df-f42e-4dd6-952e-a67ca1c68d66","Type":"ContainerStarted","Data":"1ef21eb78ec86cb83e1ffc98dc2d427858538e5cb242ac693dc96c91e4e23ef7"} Oct 30 18:17:27.099458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:27.099434 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" event={"ID":"8741f0df-f42e-4dd6-952e-a67ca1c68d66","Type":"ContainerStarted","Data":"9956890b049dd66d7ffc6c323b9afd324d513dec7e75f73120ed631d11c2744f"} Oct 30 18:17:27.134609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:27.134561 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" podStartSLOduration=2.134548603 podStartE2EDuration="2.134548603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:27.131264761 +0000 UTC m=+1143.955659053" watchObservedRunningTime="2026-10-30 18:17:27.134548603 +0000 UTC m=+1143.958942896" Oct 30 18:17:27.767430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:27.767380 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:27.831285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:27.831241 4582 patch_prober.go:29] interesting pod/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:17:27.831573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:27.831543 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:17:28.109007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:28.108917 4582 generic.go:356] "Generic (PLEG): container finished" podID="8741f0df-f42e-4dd6-952e-a67ca1c68d66" containerID="1ef21eb78ec86cb83e1ffc98dc2d427858538e5cb242ac693dc96c91e4e23ef7" exitCode=0 Oct 30 18:17:28.109399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:28.109014 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" event={"ID":"8741f0df-f42e-4dd6-952e-a67ca1c68d66","Type":"ContainerDied","Data":"1ef21eb78ec86cb83e1ffc98dc2d427858538e5cb242ac693dc96c91e4e23ef7"} Oct 30 18:17:28.835026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:28.834999 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:17:28.836206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:28.836186 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:17:29.297315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.297289 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:29.336016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.335982 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-openshift-ca\") pod \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " Oct 30 18:17:29.336273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.336060 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8741f0df-f42e-4dd6-952e-a67ca1c68d66-kube-api-access-2s7fn\" (UniqueName: \"kubernetes.io/projected/8741f0df-f42e-4dd6-952e-a67ca1c68d66-kube-api-access-2s7fn\") pod \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " Oct 30 18:17:29.336351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.336289 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-xks-ca\") pod \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\" (UID: \"8741f0df-f42e-4dd6-952e-a67ca1c68d66\") " Oct 30 18:17:29.336415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.336375 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-openshift-ca" pod "8741f0df-f42e-4dd6-952e-a67ca1c68d66" (UID: "8741f0df-f42e-4dd6-952e-a67ca1c68d66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:29.337058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.337034 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:17:29.338944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.338916 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-xks-ca" pod "8741f0df-f42e-4dd6-952e-a67ca1c68d66" (UID: "8741f0df-f42e-4dd6-952e-a67ca1c68d66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:29.339061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.339043 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8741f0df-f42e-4dd6-952e-a67ca1c68d66-kube-api-access-2s7fn" pod "8741f0df-f42e-4dd6-952e-a67ca1c68d66" (UID: "8741f0df-f42e-4dd6-952e-a67ca1c68d66"). InnerVolumeSpecName "kube-api-access-2s7fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:29.345311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.345278 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:29.438793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.438700 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s7fn\" (UniqueName: \"kubernetes.io/projected/8741f0df-f42e-4dd6-952e-a67ca1c68d66-kube-api-access-2s7fn\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:17:29.438793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:29.438731 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8741f0df-f42e-4dd6-952e-a67ca1c68d66-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:17:30.138059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:30.138033 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" Oct 30 18:17:30.138230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:30.138066 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8" event={"ID":"8741f0df-f42e-4dd6-952e-a67ca1c68d66","Type":"ContainerDied","Data":"9956890b049dd66d7ffc6c323b9afd324d513dec7e75f73120ed631d11c2744f"} Oct 30 18:17:30.138230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:30.138119 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9956890b049dd66d7ffc6c323b9afd324d513dec7e75f73120ed631d11c2744f" Oct 30 18:17:31.365618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:31.365581 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:31.563845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:31.560592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:33.028070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:33.028041 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c"] Oct 30 18:17:33.037215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:33.037191 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cbd13b-b4c5c"] Oct 30 18:17:33.101936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:33.101898 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:33.311314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:33.311246 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c51ec2-8510-46fb-927c-2723e914f06c" path="/var/lib/kubelet/pods/29c51ec2-8510-46fb-927c-2723e914f06c/volumes" Oct 30 18:17:34.273033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.272996 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l"] Oct 30 18:17:34.273659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.273646 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8741f0df-f42e-4dd6-952e-a67ca1c68d66" containerName="revoke-keys" Oct 30 18:17:34.273714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.273660 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8741f0df-f42e-4dd6-952e-a67ca1c68d66" containerName="revoke-keys" Oct 30 18:17:34.273772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.273762 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8741f0df-f42e-4dd6-952e-a67ca1c68d66" containerName="revoke-keys" Oct 30 18:17:34.276546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.276530 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.292341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.292318 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l"] Oct 30 18:17:34.295889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.295859 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:34.296017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.295895 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:17:34.296017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.295924 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:17:34.299122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.299101 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:17:34.426695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.426659 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27182be6-76d9-4fde-9228-323072ca0431-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.426949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.426932 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8vr\" (UniqueName: \"kubernetes.io/projected/27182be6-76d9-4fde-9228-323072ca0431-kube-api-access-dl8vr\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.427048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.426982 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27182be6-76d9-4fde-9228-323072ca0431-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.528067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.527991 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8vr\" (UniqueName: \"kubernetes.io/projected/27182be6-76d9-4fde-9228-323072ca0431-kube-api-access-dl8vr\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.528067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.528048 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27182be6-76d9-4fde-9228-323072ca0431-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.528281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.528132 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27182be6-76d9-4fde-9228-323072ca0431-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.528760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.528735 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27182be6-76d9-4fde-9228-323072ca0431-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.530499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.530476 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27182be6-76d9-4fde-9228-323072ca0431-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.563850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.563526 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8vr\" (UniqueName: \"kubernetes.io/projected/27182be6-76d9-4fde-9228-323072ca0431-kube-api-access-dl8vr\") pod \"maas-api-revoke-keys-e2e-api-a-4b709d-4f52l\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.590365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.590338 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:34.735914 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.735883 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:34.919986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:34.919961 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l"] Oct 30 18:17:34.922085 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:17:34.922047 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27182be6_76d9_4fde_9228_323072ca0431.slice/crio-35b892ad60c9acbd01e57d327a0f25a8bf8c409f788a8a1bfa5f3c5b53abf014 WatchSource:0}: Error finding container 35b892ad60c9acbd01e57d327a0f25a8bf8c409f788a8a1bfa5f3c5b53abf014: Status 404 returned error can't find the container with id 35b892ad60c9acbd01e57d327a0f25a8bf8c409f788a8a1bfa5f3c5b53abf014 Oct 30 18:17:35.201734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:35.201695 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" event={"ID":"27182be6-76d9-4fde-9228-323072ca0431","Type":"ContainerStarted","Data":"35b892ad60c9acbd01e57d327a0f25a8bf8c409f788a8a1bfa5f3c5b53abf014"} Oct 30 18:17:36.168815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:36.168773 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:36.213079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:36.213043 4582 generic.go:356] "Generic (PLEG): container finished" podID="27182be6-76d9-4fde-9228-323072ca0431" containerID="2faec04c0db4d3b820911bd7849f8e9f1f0b4a434fe6eb376465bb403535bd15" exitCode=0 Oct 30 18:17:36.213256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:36.213174 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" event={"ID":"27182be6-76d9-4fde-9228-323072ca0431","Type":"ContainerDied","Data":"2faec04c0db4d3b820911bd7849f8e9f1f0b4a434fe6eb376465bb403535bd15"} Oct 30 18:17:36.227427 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:17:36.227396 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:17:36.497973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:36.497934 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:37.395464 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.395428 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:37.503482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.503393 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27182be6-76d9-4fde-9228-323072ca0431-kube-api-access-dl8vr\" (UniqueName: \"kubernetes.io/projected/27182be6-76d9-4fde-9228-323072ca0431-kube-api-access-dl8vr\") pod \"27182be6-76d9-4fde-9228-323072ca0431\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " Oct 30 18:17:37.503647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.503542 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27182be6-76d9-4fde-9228-323072ca0431-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27182be6-76d9-4fde-9228-323072ca0431-maas-api-xks-ca\") pod \"27182be6-76d9-4fde-9228-323072ca0431\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " Oct 30 18:17:37.503754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.503737 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27182be6-76d9-4fde-9228-323072ca0431-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27182be6-76d9-4fde-9228-323072ca0431-maas-api-openshift-ca\") pod \"27182be6-76d9-4fde-9228-323072ca0431\" (UID: \"27182be6-76d9-4fde-9228-323072ca0431\") " Oct 30 18:17:37.504117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.504086 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27182be6-76d9-4fde-9228-323072ca0431-maas-api-openshift-ca" pod "27182be6-76d9-4fde-9228-323072ca0431" (UID: "27182be6-76d9-4fde-9228-323072ca0431"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:37.504572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.504527 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27182be6-76d9-4fde-9228-323072ca0431-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:17:37.506073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.506051 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27182be6-76d9-4fde-9228-323072ca0431-kube-api-access-dl8vr" pod "27182be6-76d9-4fde-9228-323072ca0431" (UID: "27182be6-76d9-4fde-9228-323072ca0431"). InnerVolumeSpecName "kube-api-access-dl8vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:37.506191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.506132 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27182be6-76d9-4fde-9228-323072ca0431-maas-api-xks-ca" pod "27182be6-76d9-4fde-9228-323072ca0431" (UID: "27182be6-76d9-4fde-9228-323072ca0431"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:37.605574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.605536 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27182be6-76d9-4fde-9228-323072ca0431-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:17:37.605574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:37.605564 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8vr\" (UniqueName: \"kubernetes.io/projected/27182be6-76d9-4fde-9228-323072ca0431-kube-api-access-dl8vr\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:17:38.231572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:38.231540 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:38.234297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:38.234259 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" event={"ID":"27182be6-76d9-4fde-9228-323072ca0431","Type":"ContainerDied","Data":"35b892ad60c9acbd01e57d327a0f25a8bf8c409f788a8a1bfa5f3c5b53abf014"} Oct 30 18:17:38.234458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:38.234325 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b892ad60c9acbd01e57d327a0f25a8bf8c409f788a8a1bfa5f3c5b53abf014" Oct 30 18:17:38.234458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:38.234400 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l" Oct 30 18:17:38.333084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:38.333054 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:38.442763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:38.442728 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:40.082322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:40.082295 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:41.262758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.262723 4582 generic.go:356] "Generic (PLEG): container finished" podID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerID="107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1" exitCode=0 Oct 30 18:17:41.263240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.262803 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" event={"ID":"732c9a0e-d8dc-4b8a-ab65-beae822b07da","Type":"ContainerDied","Data":"107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1"} Oct 30 18:17:41.263301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.263286 4582 scope.go:122] "RemoveContainer" containerID="107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1" Oct 30 18:17:41.264708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.264683 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:41.264759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.264751 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:41.264800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.264787 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:17:41.487072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.487042 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:17:41.948342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:41.948307 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:42.259320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:42.259231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:42.277696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:42.277662 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" event={"ID":"732c9a0e-d8dc-4b8a-ab65-beae822b07da","Type":"ContainerStarted","Data":"b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229"} Oct 30 18:17:42.909752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:42.909720 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:17:42.910650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:42.910630 4582 patch_prober.go:29] interesting pod/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:17:42.910707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:42.910675 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:17:43.623310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:43.623272 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:43.910780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:43.910699 4582 patch_prober.go:29] interesting pod/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:17:43.910780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:43.910767 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:17:44.883811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:44.881785 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:44.909878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:44.909847 4582 patch_prober.go:29] interesting pod/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:17:44.910185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:44.910138 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:17:44.982655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:44.982619 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:45.490338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:45.490294 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:45.909948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:45.909915 4582 patch_prober.go:29] interesting pod/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:17:45.910530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:45.909978 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:17:46.910167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:46.910133 4582 patch_prober.go:29] interesting pod/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:17:46.910589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:46.910187 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:17:47.427061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:47.427011 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.914133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:47.914105 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:17:47.914625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:47.914463 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:17:47.915233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:47.915214 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:17:49.060015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:49.059982 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:50.648191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:50.648158 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:52.242742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:52.242708 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:54.406434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:54.406396 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:56.080936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.080889 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n"] Oct 30 18:17:56.084442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.084412 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966dc504-pb66n"] Oct 30 18:17:56.254372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.254293 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:56.631986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.631919 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" podUID="9b802f37-fda6-4097-b93a-c5a08834f42c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:56.719962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.719906 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:56.721160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.720648 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:56.769985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.769905 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:56.832316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.829970 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:56.877939 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:56.877893 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:57.316869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:57.316836 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7f69e1-a92d-4dce-a96c-e03576d7a20b" path="/var/lib/kubelet/pods/be7f69e1-a92d-4dce-a96c-e03576d7a20b/volumes" Oct 30 18:17:57.832467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:57.832435 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:59.710805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:17:59.710774 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:01.339373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:01.339341 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:01.975568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:01.975536 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:03.034480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:03.034446 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:04.554480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:04.554448 4582 generic.go:356] "Generic (PLEG): container finished" podID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerID="baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae" exitCode=0 Oct 30 18:18:04.554981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:04.554515 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" event={"ID":"9cfbc4b9-c8ae-4011-869a-a86642c92644","Type":"ContainerDied","Data":"baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae"} Oct 30 18:18:04.554981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:04.554945 4582 scope.go:122] "RemoveContainer" containerID="baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae" Oct 30 18:18:04.556513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:04.556485 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:18:04.556574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:04.556551 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:18:04.556615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:04.556579 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:18:04.645307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:04.645263 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:05.002919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:05.002892 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:18:05.579636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:05.579601 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" event={"ID":"9cfbc4b9-c8ae-4011-869a-a86642c92644","Type":"ContainerStarted","Data":"d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1"} Oct 30 18:18:05.823674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:05.823640 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:18:05.824471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:05.824448 4582 patch_prober.go:29] interesting pod/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:18:05.824585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:05.824493 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:18:06.028722 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:06.028691 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx"] Oct 30 18:18:06.033376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:06.033346 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16f5427-cdncx"] Oct 30 18:18:06.225698 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:18:06.225661 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:18:06.234570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:06.234517 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:06.823889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:06.823853 4582 patch_prober.go:29] interesting pod/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:18:06.824455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:06.824419 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:18:07.301264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:07.301232 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f4c4cf-2eb2-4490-8719-795ea7f3ff53" path="/var/lib/kubelet/pods/c8f4c4cf-2eb2-4490-8719-795ea7f3ff53/volumes" Oct 30 18:18:07.746846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:07.742011 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:07.823981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:07.823945 4582 patch_prober.go:29] interesting pod/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:18:07.824448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:07.824023 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:18:08.823621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:08.823588 4582 patch_prober.go:29] interesting pod/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:18:08.823810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:08.823669 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:18:09.426154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:09.426118 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:09.823580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:09.823473 4582 patch_prober.go:29] interesting pod/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:18:09.823580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:09.823534 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:18:10.685701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:10.685661 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:10.785475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:10.785437 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:10.852932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:10.852898 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:18:10.854364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:10.854338 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:18:10.855352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:10.855324 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:18:10.942748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:10.942671 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:12.628500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:12.628462 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.330705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.330666 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz"] Oct 30 18:18:13.331625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.331597 4582 cpu_manager.go:414] "containerMap: removing container" podUID="27182be6-76d9-4fde-9228-323072ca0431" containerName="revoke-keys" Oct 30 18:18:13.331625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.331622 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27182be6-76d9-4fde-9228-323072ca0431" containerName="revoke-keys" Oct 30 18:18:13.331878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.331865 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="27182be6-76d9-4fde-9228-323072ca0431" containerName="revoke-keys" Oct 30 18:18:13.335092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.335075 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.350725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.350697 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz"] Oct 30 18:18:13.354319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.354293 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:13.354319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.354312 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:18:13.354466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.354350 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:18:13.356272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.356254 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:18:13.432854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.432793 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.433036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.432981 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.433110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.433082 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zmf\" (UniqueName: \"kubernetes.io/projected/03ce866e-c500-4f6a-851c-bd15ce2ac83d-kube-api-access-t6zmf\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.535001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.534967 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.535152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.535094 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.535217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.535195 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zmf\" (UniqueName: \"kubernetes.io/projected/03ce866e-c500-4f6a-851c-bd15ce2ac83d-kube-api-access-t6zmf\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.535734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.535709 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.537372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.537348 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.551125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.551100 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zmf\" (UniqueName: \"kubernetes.io/projected/03ce866e-c500-4f6a-851c-bd15ce2ac83d-kube-api-access-t6zmf\") pod \"maas-api-revoke-keys-e2e-api-a-f15587-86hzz\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.598371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.598304 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.650946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.650921 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:13.950551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:13.950528 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz"] Oct 30 18:18:13.952557 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:18:13.952525 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ce866e_c500_4f6a_851c_bd15ce2ac83d.slice/crio-9997738894a4d9410d6716dd524938f64ac10a1949ef0050418802a1fd0fba2f WatchSource:0}: Error finding container 9997738894a4d9410d6716dd524938f64ac10a1949ef0050418802a1fd0fba2f: Status 404 returned error can't find the container with id 9997738894a4d9410d6716dd524938f64ac10a1949ef0050418802a1fd0fba2f Oct 30 18:18:14.384309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:14.384280 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:14.654308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:14.654275 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" event={"ID":"03ce866e-c500-4f6a-851c-bd15ce2ac83d","Type":"ContainerStarted","Data":"978e8c8d790aec07ec9b6a20cbe1cdb744876e130d14906d97ec5010509862e8"} Oct 30 18:18:14.654772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:14.654327 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" event={"ID":"03ce866e-c500-4f6a-851c-bd15ce2ac83d","Type":"ContainerStarted","Data":"9997738894a4d9410d6716dd524938f64ac10a1949ef0050418802a1fd0fba2f"} Oct 30 18:18:14.687655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:14.687610 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" podStartSLOduration=1.6875989310000001 podStartE2EDuration="1.687598931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:14.68707278 +0000 UTC m=+1191.511467071" watchObservedRunningTime="2026-10-30 18:18:14.687598931 +0000 UTC m=+1191.511993222" Oct 30 18:18:15.663653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:15.663621 4582 generic.go:356] "Generic (PLEG): container finished" podID="03ce866e-c500-4f6a-851c-bd15ce2ac83d" containerID="978e8c8d790aec07ec9b6a20cbe1cdb744876e130d14906d97ec5010509862e8" exitCode=0 Oct 30 18:18:15.664109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:15.663711 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" event={"ID":"03ce866e-c500-4f6a-851c-bd15ce2ac83d","Type":"ContainerDied","Data":"978e8c8d790aec07ec9b6a20cbe1cdb744876e130d14906d97ec5010509862e8"} Oct 30 18:18:15.773070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:15.773032 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:16.320761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.320723 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:16.635812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.635782 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:16.740319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.739707 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l"] Oct 30 18:18:16.751857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.749950 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4b709d-4f52l"] Oct 30 18:18:16.867857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.867812 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:16.885299 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.885263 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:16.989547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.989468 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-xks-ca\") pod \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " Oct 30 18:18:16.989547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.989510 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ce866e-c500-4f6a-851c-bd15ce2ac83d-kube-api-access-t6zmf\" (UniqueName: \"kubernetes.io/projected/03ce866e-c500-4f6a-851c-bd15ce2ac83d-kube-api-access-t6zmf\") pod \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " Oct 30 18:18:16.989750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.989709 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-openshift-ca\") pod \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\" (UID: \"03ce866e-c500-4f6a-851c-bd15ce2ac83d\") " Oct 30 18:18:16.990081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.990052 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-openshift-ca" pod "03ce866e-c500-4f6a-851c-bd15ce2ac83d" (UID: "03ce866e-c500-4f6a-851c-bd15ce2ac83d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:16.990373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.990354 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.991688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.991659 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-xks-ca" pod "03ce866e-c500-4f6a-851c-bd15ce2ac83d" (UID: "03ce866e-c500-4f6a-851c-bd15ce2ac83d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:16.991982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:16.991962 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ce866e-c500-4f6a-851c-bd15ce2ac83d-kube-api-access-t6zmf" pod "03ce866e-c500-4f6a-851c-bd15ce2ac83d" (UID: "03ce866e-c500-4f6a-851c-bd15ce2ac83d"). InnerVolumeSpecName "kube-api-access-t6zmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:17.093440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:17.093405 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce866e-c500-4f6a-851c-bd15ce2ac83d-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:17.093440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:17.093440 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6zmf\" (UniqueName: \"kubernetes.io/projected/03ce866e-c500-4f6a-851c-bd15ce2ac83d-kube-api-access-t6zmf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:17.227455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:17.227421 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:17.302772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:17.302702 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27182be6-76d9-4fde-9228-323072ca0431" path="/var/lib/kubelet/pods/27182be6-76d9-4fde-9228-323072ca0431/volumes" Oct 30 18:18:17.681645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:17.681615 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" Oct 30 18:18:17.681832 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:17.681615 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz" event={"ID":"03ce866e-c500-4f6a-851c-bd15ce2ac83d","Type":"ContainerDied","Data":"9997738894a4d9410d6716dd524938f64ac10a1949ef0050418802a1fd0fba2f"} Oct 30 18:18:17.681832 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:17.681735 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9997738894a4d9410d6716dd524938f64ac10a1949ef0050418802a1fd0fba2f" Oct 30 18:18:18.537675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:18.537643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:20.036514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.036479 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:20.554516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.554485 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7619-cc48c5"] Oct 30 18:18:20.555187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.555173 4582 cpu_manager.go:414] "containerMap: removing container" podUID="03ce866e-c500-4f6a-851c-bd15ce2ac83d" containerName="revoke-keys" Oct 30 18:18:20.555250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.555188 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ce866e-c500-4f6a-851c-bd15ce2ac83d" containerName="revoke-keys" Oct 30 18:18:20.555310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.555301 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ce866e-c500-4f6a-851c-bd15ce2ac83d" containerName="revoke-keys" Oct 30 18:18:20.558344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.558329 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:20.575411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.575382 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7619-cc48c5"] Oct 30 18:18:20.578207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.578185 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r2spd\"" Oct 30 18:18:20.641190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.641160 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnrqc\" (UniqueName: \"kubernetes.io/projected/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832-kube-api-access-qnrqc\") pod \"test-curl-7619-cc48c5\" (UID: \"7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832\") " pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:20.742170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.742144 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrqc\" (UniqueName: \"kubernetes.io/projected/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832-kube-api-access-qnrqc\") pod \"test-curl-7619-cc48c5\" (UID: \"7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832\") " pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:20.758679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.758649 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrqc\" (UniqueName: \"kubernetes.io/projected/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832-kube-api-access-qnrqc\") pod \"test-curl-7619-cc48c5\" (UID: \"7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832\") " pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:20.871005 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:20.870976 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:21.171419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:21.171391 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7619-cc48c5"] Oct 30 18:18:21.173181 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:18:21.173152 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d8ebbc7_14b5_4cdb_a5ab_58f0838a9832.slice/crio-580e40f83bc517223da907e8887518e93368f1b310e826ce97a8609889988b47 WatchSource:0}: Error finding container 580e40f83bc517223da907e8887518e93368f1b310e826ce97a8609889988b47: Status 404 returned error can't find the container with id 580e40f83bc517223da907e8887518e93368f1b310e826ce97a8609889988b47 Oct 30 18:18:21.175078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:21.175060 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:18:21.568130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:21.568044 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:21.719484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:21.719447 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7619-cc48c5" event={"ID":"7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832","Type":"ContainerStarted","Data":"580e40f83bc517223da907e8887518e93368f1b310e826ce97a8609889988b47"} Oct 30 18:18:23.313081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:23.313037 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.421385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:23.421314 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.721352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:23.721315 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.738301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:23.738264 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7619-cc48c5" event={"ID":"7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832","Type":"ContainerStarted","Data":"a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3"} Oct 30 18:18:23.816962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:23.812721 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7619-cc48c5" podStartSLOduration=1.742088686 podStartE2EDuration="3.812699431s" totalImagesPullingTime="2.070610745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:23.779581753 +0000 UTC m=+1200.603976049" watchObservedRunningTime="2026-10-30 18:18:23.812699431 +0000 UTC m=+1200.637093726" Oct 30 18:18:24.039922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.039835 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:24.129099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.129062 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:24.353338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.353259 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf"] Oct 30 18:18:24.357409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.357385 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.375161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.375139 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:18:24.384776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.384748 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:24.384776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.384757 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:18:24.385061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.384850 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:18:24.392185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.392158 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf"] Oct 30 18:18:24.438349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.438319 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.438551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.438472 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.438775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.438751 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdtf\" (UniqueName: \"kubernetes.io/projected/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-kube-api-access-4vdtf\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.540226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.540186 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdtf\" (UniqueName: \"kubernetes.io/projected/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-kube-api-access-4vdtf\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.540431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.540334 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.540503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.540462 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.541588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.541559 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.543198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.543174 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.566262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.566234 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdtf\" (UniqueName: \"kubernetes.io/projected/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-kube-api-access-4vdtf\") pod \"maas-api-revoke-keys-e2e-api-b-b33a49-g78tf\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:24.656299 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.656263 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7619-cc48c5"] Oct 30 18:18:24.662502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.662473 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7619-cc48c5"] Oct 30 18:18:24.672578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:24.672544 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:25.119942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.119915 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf"] Oct 30 18:18:25.121096 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:18:25.121070 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c5e3fb_e712_48c1_a8b0_ad2fadb45b5f.slice/crio-45810811321f8eb6961a39d173736faa5b2c2a8dbd4926a63d96ada52c56389d WatchSource:0}: Error finding container 45810811321f8eb6961a39d173736faa5b2c2a8dbd4926a63d96ada52c56389d: Status 404 returned error can't find the container with id 45810811321f8eb6961a39d173736faa5b2c2a8dbd4926a63d96ada52c56389d Oct 30 18:18:25.412278 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.412244 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69"] Oct 30 18:18:25.413030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.413002 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" containerName="test-curl-7619-cc48c5" Oct 30 18:18:25.413086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.413034 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" containerName="test-curl-7619-cc48c5" Oct 30 18:18:25.413160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.413151 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" containerName="test-curl-7619-cc48c5" Oct 30 18:18:25.417205 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.417185 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.433329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.433308 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69"] Oct 30 18:18:25.506435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.506408 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.506604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.506528 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.506750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.506722 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv58b\" (UniqueName: \"kubernetes.io/projected/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-kube-api-access-cv58b\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.507387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.507369 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.607927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.607889 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv58b\" (UniqueName: \"kubernetes.io/projected/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-kube-api-access-cv58b\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.608130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.607997 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.608130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.608091 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.608892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.608868 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.610365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.610346 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.627854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.627782 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv58b\" (UniqueName: \"kubernetes.io/projected/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-kube-api-access-cv58b\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.731258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.731183 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:25.873812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.873130 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" event={"ID":"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f","Type":"ContainerStarted","Data":"f2a84763d56d1dc61f7dc2d89b6b7060b086e967ecb2589d8a8ce511d1674fdd"} Oct 30 18:18:25.873812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.873199 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" event={"ID":"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f","Type":"ContainerStarted","Data":"45810811321f8eb6961a39d173736faa5b2c2a8dbd4926a63d96ada52c56389d"} Oct 30 18:18:25.873812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.873241 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7619-cc48c5" podUID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" containerName="test-curl-7619-cc48c5" containerID="cri-o://a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3" gracePeriod=2 Oct 30 18:18:25.910481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:25.910428 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" podStartSLOduration=1.910413069 podStartE2EDuration="1.910413069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:25.910118967 +0000 UTC m=+1202.734513285" watchObservedRunningTime="2026-10-30 18:18:25.910413069 +0000 UTC m=+1202.734807364" Oct 30 18:18:26.038676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:26.038647 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69"] Oct 30 18:18:26.040648 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:18:26.040621 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c31dfc_92b1_46f2_a6e5_cfb7e5839c89.slice/crio-4b0e6107933a92f5b97bb53d61920ea8698222c9177cc0cd994bd6bd9b8ee441 WatchSource:0}: Error finding container 4b0e6107933a92f5b97bb53d61920ea8698222c9177cc0cd994bd6bd9b8ee441: Status 404 returned error can't find the container with id 4b0e6107933a92f5b97bb53d61920ea8698222c9177cc0cd994bd6bd9b8ee441 Oct 30 18:18:26.887228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:26.887187 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" event={"ID":"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89","Type":"ContainerStarted","Data":"32ea10073df8e839b23ea4fa2e54aa0ee3ed2907c76a8ad662adcd2aa3b953f7"} Oct 30 18:18:26.887697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:26.887277 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" event={"ID":"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89","Type":"ContainerStarted","Data":"4b0e6107933a92f5b97bb53d61920ea8698222c9177cc0cd994bd6bd9b8ee441"} Oct 30 18:18:26.889018 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:26.888993 4582 generic.go:356] "Generic (PLEG): container finished" podID="80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" containerID="f2a84763d56d1dc61f7dc2d89b6b7060b086e967ecb2589d8a8ce511d1674fdd" exitCode=0 Oct 30 18:18:26.889119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:26.889045 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" event={"ID":"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f","Type":"ContainerDied","Data":"f2a84763d56d1dc61f7dc2d89b6b7060b086e967ecb2589d8a8ce511d1674fdd"} Oct 30 18:18:27.340342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:27.340267 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:27.899295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:27.899265 4582 generic.go:356] "Generic (PLEG): container finished" podID="97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" containerID="32ea10073df8e839b23ea4fa2e54aa0ee3ed2907c76a8ad662adcd2aa3b953f7" exitCode=0 Oct 30 18:18:27.899605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:27.899340 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" event={"ID":"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89","Type":"ContainerDied","Data":"32ea10073df8e839b23ea4fa2e54aa0ee3ed2907c76a8ad662adcd2aa3b953f7"} Oct 30 18:18:28.117686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.117661 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:28.121528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.121504 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:28.144545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.144517 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-openshift-ca\") pod \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " Oct 30 18:18:28.144689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.144645 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-xks-ca\") pod \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " Oct 30 18:18:28.144754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.144689 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-kube-api-access-cv58b\" (UniqueName: \"kubernetes.io/projected/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-kube-api-access-cv58b\") pod \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " Oct 30 18:18:28.144754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.144717 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-kube-api-access-4vdtf\" (UniqueName: \"kubernetes.io/projected/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-kube-api-access-4vdtf\") pod \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " Oct 30 18:18:28.144754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.144742 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-xks-ca\") pod \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\" (UID: \"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f\") " Oct 30 18:18:28.144928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.144781 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-openshift-ca\") pod \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\" (UID: \"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89\") " Oct 30 18:18:28.144928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.144875 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-openshift-ca" pod "80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" (UID: "80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:28.145474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.145447 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-openshift-ca" pod "97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" (UID: "97c31dfc-92b1-46f2-a6e5-cfb7e5839c89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:28.145565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.145518 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.147445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.147416 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-kube-api-access-cv58b" pod "97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" (UID: "97c31dfc-92b1-46f2-a6e5-cfb7e5839c89"). InnerVolumeSpecName "kube-api-access-cv58b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:28.147589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.147449 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-xks-ca" pod "80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" (UID: "80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:28.147589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.147488 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-xks-ca" pod "97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" (UID: "97c31dfc-92b1-46f2-a6e5-cfb7e5839c89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:28.148046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.148019 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-kube-api-access-4vdtf" pod "80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" (UID: "80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f"). InnerVolumeSpecName "kube-api-access-4vdtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:28.156682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.156659 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:28.246240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.246211 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832-kube-api-access-qnrqc\" (UniqueName: \"kubernetes.io/projected/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832-kube-api-access-qnrqc\") pod \"7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832\" (UID: \"7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832\") " Oct 30 18:18:28.246610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.246597 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.246652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.246613 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv58b\" (UniqueName: \"kubernetes.io/projected/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-kube-api-access-cv58b\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.246652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.246623 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vdtf\" (UniqueName: \"kubernetes.io/projected/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-kube-api-access-4vdtf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.246652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.246632 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.246652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.246640 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.248209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.248188 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832-kube-api-access-qnrqc" pod "7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" (UID: "7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832"). InnerVolumeSpecName "kube-api-access-qnrqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:28.348147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.348114 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnrqc\" (UniqueName: \"kubernetes.io/projected/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832-kube-api-access-qnrqc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.908430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.908403 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" Oct 30 18:18:28.908865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.908402 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69" event={"ID":"97c31dfc-92b1-46f2-a6e5-cfb7e5839c89","Type":"ContainerDied","Data":"4b0e6107933a92f5b97bb53d61920ea8698222c9177cc0cd994bd6bd9b8ee441"} Oct 30 18:18:28.908865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.908548 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0e6107933a92f5b97bb53d61920ea8698222c9177cc0cd994bd6bd9b8ee441" Oct 30 18:18:28.909762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.909736 4582 generic.go:356] "Generic (PLEG): container finished" podID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" containerID="a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3" exitCode=137 Oct 30 18:18:28.909920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.909778 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:28.909920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.909829 4582 scope.go:122] "RemoveContainer" containerID="a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3" Oct 30 18:18:28.911550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.911514 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" event={"ID":"80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f","Type":"ContainerDied","Data":"45810811321f8eb6961a39d173736faa5b2c2a8dbd4926a63d96ada52c56389d"} Oct 30 18:18:28.911550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.911538 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf" Oct 30 18:18:28.911684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.911556 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45810811321f8eb6961a39d173736faa5b2c2a8dbd4926a63d96ada52c56389d" Oct 30 18:18:28.920944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.920863 4582 scope.go:122] "RemoveContainer" containerID="a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3" Oct 30 18:18:28.921220 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:18:28.921196 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3\": container with ID starting with a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3 not found: ID does not exist" containerID="a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3" Oct 30 18:18:28.921311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:28.921223 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3"} err="failed to get container status \"a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3\": rpc error: code = NotFound desc = could not find container \"a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3\": container with ID starting with a07fc06099623965a7fcc587d24e6309417a629877265dd4e0374474e689d2c3 not found: ID does not exist" Oct 30 18:18:28.932413 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:18:28.932390 4582 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7619-cc48c5\" is forbidden: User \"system:node:ip-10-0-133-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-124.ec2.internal' and this object" podUID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:28.943023 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:18:28.942999 4582 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7619-cc48c5\" is forbidden: User \"system:node:ip-10-0-133-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-124.ec2.internal' and this object" podUID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" pod="opendatahub/test-curl-7619-cc48c5" Oct 30 18:18:29.094130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:29.094085 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:29.107236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:29.107201 4582 scope.go:122] "RemoveContainer" containerID="b3188e559adfc59d1ce9c64cabc08430aef499d52cedda2414e7c35006b34080" Oct 30 18:18:29.118615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:29.118592 4582 scope.go:122] "RemoveContainer" containerID="5f2a8234f45fd6e1f04740da5c6b7b1f08aefbd853006bdcfb4224287dcb0282" Oct 30 18:18:29.130159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:29.130138 4582 scope.go:122] "RemoveContainer" containerID="80b7ce92357b5997873777f3b3c1854a128dcb70b40bd8d0b48b253bb4b0dff9" Oct 30 18:18:29.141737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:29.141715 4582 scope.go:122] "RemoveContainer" containerID="3e02a0f500fb2e51ae05ed60f912b709387d5a7affbb4c1711c1e7a8f112f774" Oct 30 18:18:29.150915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:29.150893 4582 scope.go:122] "RemoveContainer" containerID="e7b10939366f3888dbfca90f794f6b98697c7fa7419e9586689a46f25318d9f2" Oct 30 18:18:29.301678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:29.301647 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832" path="/var/lib/kubelet/pods/7d8ebbc7-14b5-4cdb-a5ab-58f0838a9832/volumes" Oct 30 18:18:30.752515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:30.752479 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:32.044938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:32.044906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:32.423356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:32.423322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:34.115268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:34.115234 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:34.975102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:34.975068 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.092786 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:35.092742 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.824139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:35.824107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.947013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:35.946967 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:36.225546 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:18:36.225510 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:18:37.538148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:37.538112 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:39.262805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:39.262771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:40.778279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:40.778246 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:41.554147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:41.554073 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:41.694511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:41.694481 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:41.753323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:41.753291 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz"] Oct 30 18:18:41.763869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:41.763838 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f15587-86hzz"] Oct 30 18:18:41.795381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:41.795342 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:42.218836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:42.218796 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:43.301459 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:43.301424 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ce866e-c500-4f6a-851c-bd15ce2ac83d" path="/var/lib/kubelet/pods/03ce866e-c500-4f6a-851c-bd15ce2ac83d/volumes" Oct 30 18:18:43.563341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:43.563261 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:44.698942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.698894 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d"] Oct 30 18:18:44.699836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.699805 4582 cpu_manager.go:414] "containerMap: removing container" podUID="97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" containerName="revoke-keys" Oct 30 18:18:44.699894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.699838 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" containerName="revoke-keys" Oct 30 18:18:44.699894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.699868 4582 cpu_manager.go:414] "containerMap: removing container" podUID="80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" containerName="revoke-keys" Oct 30 18:18:44.699894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.699873 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" containerName="revoke-keys" Oct 30 18:18:44.700009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.699992 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" containerName="revoke-keys" Oct 30 18:18:44.700050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.700010 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" containerName="revoke-keys" Oct 30 18:18:44.704374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.704353 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.720173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.720147 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d"] Oct 30 18:18:44.724689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.724664 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:18:44.724801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.724766 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:44.724801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.724782 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:18:44.727720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.727695 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:18:44.796980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.796951 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.797174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.797007 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.797364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.797346 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrgb\" (UniqueName: \"kubernetes.io/projected/bc0a411c-af6b-418a-b128-917efdf2c602-kube-api-access-pkrgb\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.898641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.898606 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgb\" (UniqueName: \"kubernetes.io/projected/bc0a411c-af6b-418a-b128-917efdf2c602-kube-api-access-pkrgb\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.898809 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.898666 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.898809 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.898695 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.899390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.899365 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.901053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.901031 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.914970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.914941 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrgb\" (UniqueName: \"kubernetes.io/projected/bc0a411c-af6b-418a-b128-917efdf2c602-kube-api-access-pkrgb\") pod \"maas-api-revoke-sub-341e8382-ddw6d\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:44.941847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.941794 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl"] Oct 30 18:18:44.945096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.945079 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:44.963782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:44.963735 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl"] Oct 30 18:18:45.000319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.000279 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.000489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.000326 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hj2\" (UniqueName: \"kubernetes.io/projected/f7625e37-12c0-4366-950a-ae4ab7fa4350-kube-api-access-f8hj2\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.000605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.000587 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.019891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.019806 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:45.102558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.102444 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.102722 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.102646 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.102722 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.102682 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hj2\" (UniqueName: \"kubernetes.io/projected/f7625e37-12c0-4366-950a-ae4ab7fa4350-kube-api-access-f8hj2\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.103716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.103675 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.106044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.106009 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.120296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.120260 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hj2\" (UniqueName: \"kubernetes.io/projected/f7625e37-12c0-4366-950a-ae4ab7fa4350-kube-api-access-f8hj2\") pod \"maas-api-revoke-sub-73668924-lpbsl\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.254905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.254796 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:45.259813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.259788 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:45.339645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.339613 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d"] Oct 30 18:18:45.341023 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:18:45.340989 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0a411c_af6b_418a_b128_917efdf2c602.slice/crio-6a747089ea242bc81c423e89f8ee41692c8d79e84158c95ce928ecd290cf3d29 WatchSource:0}: Error finding container 6a747089ea242bc81c423e89f8ee41692c8d79e84158c95ce928ecd290cf3d29: Status 404 returned error can't find the container with id 6a747089ea242bc81c423e89f8ee41692c8d79e84158c95ce928ecd290cf3d29 Oct 30 18:18:45.576941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:45.576920 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl"] Oct 30 18:18:45.577572 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:18:45.577546 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7625e37_12c0_4366_950a_ae4ab7fa4350.slice/crio-b2b01b3a29f5f54fdfe27aee5f24308a9a389d53c30a5deab19ec07de255eda8 WatchSource:0}: Error finding container b2b01b3a29f5f54fdfe27aee5f24308a9a389d53c30a5deab19ec07de255eda8: Status 404 returned error can't find the container with id b2b01b3a29f5f54fdfe27aee5f24308a9a389d53c30a5deab19ec07de255eda8 Oct 30 18:18:46.067903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:46.067865 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" event={"ID":"f7625e37-12c0-4366-950a-ae4ab7fa4350","Type":"ContainerStarted","Data":"9ad6ca9956f14d65035a3c882f4de30694a40b1eaf4240a8e69647508c2c5a6d"} Oct 30 18:18:46.068362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:46.067927 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" event={"ID":"f7625e37-12c0-4366-950a-ae4ab7fa4350","Type":"ContainerStarted","Data":"b2b01b3a29f5f54fdfe27aee5f24308a9a389d53c30a5deab19ec07de255eda8"} Oct 30 18:18:46.069577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:46.069550 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" event={"ID":"bc0a411c-af6b-418a-b128-917efdf2c602","Type":"ContainerStarted","Data":"e8d9ae517ff30f8db7c3e964ebf6505d4d06d2accd4e3b7dca6692836d0bd379"} Oct 30 18:18:46.069704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:46.069596 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" event={"ID":"bc0a411c-af6b-418a-b128-917efdf2c602","Type":"ContainerStarted","Data":"6a747089ea242bc81c423e89f8ee41692c8d79e84158c95ce928ecd290cf3d29"} Oct 30 18:18:46.108917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:46.108831 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" podStartSLOduration=2.10880352 podStartE2EDuration="2.10880352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:46.103292398 +0000 UTC m=+1222.927686710" watchObservedRunningTime="2026-10-30 18:18:46.10880352 +0000 UTC m=+1222.933197812" Oct 30 18:18:46.142961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:46.142892 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" podStartSLOduration=2.142873205 podStartE2EDuration="2.142873205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:46.13803077 +0000 UTC m=+1222.962425062" watchObservedRunningTime="2026-10-30 18:18:46.142873205 +0000 UTC m=+1222.967267499" Oct 30 18:18:46.942213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:46.942177 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:47.081911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:47.081876 4582 generic.go:356] "Generic (PLEG): container finished" podID="f7625e37-12c0-4366-950a-ae4ab7fa4350" containerID="9ad6ca9956f14d65035a3c882f4de30694a40b1eaf4240a8e69647508c2c5a6d" exitCode=0 Oct 30 18:18:47.082373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:47.081915 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" event={"ID":"f7625e37-12c0-4366-950a-ae4ab7fa4350","Type":"ContainerDied","Data":"9ad6ca9956f14d65035a3c882f4de30694a40b1eaf4240a8e69647508c2c5a6d"} Oct 30 18:18:47.089242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:47.084364 4582 generic.go:356] "Generic (PLEG): container finished" podID="bc0a411c-af6b-418a-b128-917efdf2c602" containerID="e8d9ae517ff30f8db7c3e964ebf6505d4d06d2accd4e3b7dca6692836d0bd379" exitCode=0 Oct 30 18:18:47.089242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:47.084509 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" event={"ID":"bc0a411c-af6b-418a-b128-917efdf2c602","Type":"ContainerDied","Data":"e8d9ae517ff30f8db7c3e964ebf6505d4d06d2accd4e3b7dca6692836d0bd379"} Oct 30 18:18:48.196473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.196439 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:48.289646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.289619 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:48.301222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.301117 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:48.346297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.346221 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:48.348436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.348398 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-openshift-ca\") pod \"bc0a411c-af6b-418a-b128-917efdf2c602\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " Oct 30 18:18:48.348573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.348469 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0a411c-af6b-418a-b128-917efdf2c602-kube-api-access-pkrgb\" (UniqueName: \"kubernetes.io/projected/bc0a411c-af6b-418a-b128-917efdf2c602-kube-api-access-pkrgb\") pod \"bc0a411c-af6b-418a-b128-917efdf2c602\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " Oct 30 18:18:48.348771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.348751 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-xks-ca\") pod \"bc0a411c-af6b-418a-b128-917efdf2c602\" (UID: \"bc0a411c-af6b-418a-b128-917efdf2c602\") " Oct 30 18:18:48.349242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.349216 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-openshift-ca" pod "bc0a411c-af6b-418a-b128-917efdf2c602" (UID: "bc0a411c-af6b-418a-b128-917efdf2c602"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:48.349933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.349908 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:48.351356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.351331 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-xks-ca" pod "bc0a411c-af6b-418a-b128-917efdf2c602" (UID: "bc0a411c-af6b-418a-b128-917efdf2c602"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:48.351450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.351363 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0a411c-af6b-418a-b128-917efdf2c602-kube-api-access-pkrgb" pod "bc0a411c-af6b-418a-b128-917efdf2c602" (UID: "bc0a411c-af6b-418a-b128-917efdf2c602"). InnerVolumeSpecName "kube-api-access-pkrgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:48.450801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.450773 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-xks-ca\") pod \"f7625e37-12c0-4366-950a-ae4ab7fa4350\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " Oct 30 18:18:48.451036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.450849 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-openshift-ca\") pod \"f7625e37-12c0-4366-950a-ae4ab7fa4350\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " Oct 30 18:18:48.451036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.450915 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7625e37-12c0-4366-950a-ae4ab7fa4350-kube-api-access-f8hj2\" (UniqueName: \"kubernetes.io/projected/f7625e37-12c0-4366-950a-ae4ab7fa4350-kube-api-access-f8hj2\") pod \"f7625e37-12c0-4366-950a-ae4ab7fa4350\" (UID: \"f7625e37-12c0-4366-950a-ae4ab7fa4350\") " Oct 30 18:18:48.451226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.451202 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-openshift-ca" pod "f7625e37-12c0-4366-950a-ae4ab7fa4350" (UID: "f7625e37-12c0-4366-950a-ae4ab7fa4350"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:48.451609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.451590 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkrgb\" (UniqueName: \"kubernetes.io/projected/bc0a411c-af6b-418a-b128-917efdf2c602-kube-api-access-pkrgb\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:48.451686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.451615 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0a411c-af6b-418a-b128-917efdf2c602-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:48.451686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.451629 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:48.452858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.452836 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-xks-ca" pod "f7625e37-12c0-4366-950a-ae4ab7fa4350" (UID: "f7625e37-12c0-4366-950a-ae4ab7fa4350"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:48.452966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.452892 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7625e37-12c0-4366-950a-ae4ab7fa4350-kube-api-access-f8hj2" pod "f7625e37-12c0-4366-950a-ae4ab7fa4350" (UID: "f7625e37-12c0-4366-950a-ae4ab7fa4350"). InnerVolumeSpecName "kube-api-access-f8hj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:48.522879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.515946 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2"] Oct 30 18:18:48.522879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.516171 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" containerID="cri-o://80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8" gracePeriod=30 Oct 30 18:18:48.534012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.533941 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:48.554059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.554033 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8hj2\" (UniqueName: \"kubernetes.io/projected/f7625e37-12c0-4366-950a-ae4ab7fa4350-kube-api-access-f8hj2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:48.554059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.554057 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7625e37-12c0-4366-950a-ae4ab7fa4350-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:48.717841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:48.695211 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:49.102572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.102479 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" Oct 30 18:18:49.102572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.102498 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d" event={"ID":"bc0a411c-af6b-418a-b128-917efdf2c602","Type":"ContainerDied","Data":"6a747089ea242bc81c423e89f8ee41692c8d79e84158c95ce928ecd290cf3d29"} Oct 30 18:18:49.102572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.102545 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a747089ea242bc81c423e89f8ee41692c8d79e84158c95ce928ecd290cf3d29" Oct 30 18:18:49.104261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.104218 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" Oct 30 18:18:49.104261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.104238 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl" event={"ID":"f7625e37-12c0-4366-950a-ae4ab7fa4350","Type":"ContainerDied","Data":"b2b01b3a29f5f54fdfe27aee5f24308a9a389d53c30a5deab19ec07de255eda8"} Oct 30 18:18:49.104435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.104282 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2b01b3a29f5f54fdfe27aee5f24308a9a389d53c30a5deab19ec07de255eda8" Oct 30 18:18:49.169227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.169192 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9"] Oct 30 18:18:49.170164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.170135 4582 cpu_manager.go:414] "containerMap: removing container" podUID="bc0a411c-af6b-418a-b128-917efdf2c602" containerName="revoke-keys" Oct 30 18:18:49.170164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.170159 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0a411c-af6b-418a-b128-917efdf2c602" containerName="revoke-keys" Oct 30 18:18:49.170378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.170239 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f7625e37-12c0-4366-950a-ae4ab7fa4350" containerName="revoke-keys" Oct 30 18:18:49.170378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.170248 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7625e37-12c0-4366-950a-ae4ab7fa4350" containerName="revoke-keys" Oct 30 18:18:49.170493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.170423 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7625e37-12c0-4366-950a-ae4ab7fa4350" containerName="revoke-keys" Oct 30 18:18:49.170493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.170443 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0a411c-af6b-418a-b128-917efdf2c602" containerName="revoke-keys" Oct 30 18:18:49.173669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.173646 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.211852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.201808 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:49.211852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.201912 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:18:49.211852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.202000 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:18:49.211852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.202088 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:18:49.216567 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.216537 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9"] Oct 30 18:18:49.314338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.314299 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zdln\" (UniqueName: \"kubernetes.io/projected/8c13b847-7063-4a20-ad97-48061bb6b8eb-kube-api-access-7zdln\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.314474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.314458 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.314585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.314570 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.415769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.415732 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.415942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.415885 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.416047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.416032 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zdln\" (UniqueName: \"kubernetes.io/projected/8c13b847-7063-4a20-ad97-48061bb6b8eb-kube-api-access-7zdln\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.416446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.416425 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.418314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.418294 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.432723 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.432703 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zdln\" (UniqueName: \"kubernetes.io/projected/8c13b847-7063-4a20-ad97-48061bb6b8eb-kube-api-access-7zdln\") pod \"maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.514479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.514446 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:49.814656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:49.814632 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9"] Oct 30 18:18:49.815271 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:18:49.815248 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c13b847_7063_4a20_ad97_48061bb6b8eb.slice/crio-72525b04b463e084d8b3867932614a7c6fdc1c938a48141460e52cfeaa44a331 WatchSource:0}: Error finding container 72525b04b463e084d8b3867932614a7c6fdc1c938a48141460e52cfeaa44a331: Status 404 returned error can't find the container with id 72525b04b463e084d8b3867932614a7c6fdc1c938a48141460e52cfeaa44a331 Oct 30 18:18:50.113026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:50.112954 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" event={"ID":"8c13b847-7063-4a20-ad97-48061bb6b8eb","Type":"ContainerStarted","Data":"72525b04b463e084d8b3867932614a7c6fdc1c938a48141460e52cfeaa44a331"} Oct 30 18:18:50.605886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:50.605846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:51.122381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:51.122346 4582 generic.go:356] "Generic (PLEG): container finished" podID="8c13b847-7063-4a20-ad97-48061bb6b8eb" containerID="3a66a26ee783c58a1d8d2480cb0be731ba7a9f5bb1dac82eb57a944fad842212" exitCode=0 Oct 30 18:18:51.122549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:51.122429 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" event={"ID":"8c13b847-7063-4a20-ad97-48061bb6b8eb","Type":"ContainerDied","Data":"3a66a26ee783c58a1d8d2480cb0be731ba7a9f5bb1dac82eb57a944fad842212"} Oct 30 18:18:52.294694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.294668 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:52.347454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.347412 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:52.462095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.461957 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-xks-ca\") pod \"8c13b847-7063-4a20-ad97-48061bb6b8eb\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " Oct 30 18:18:52.462340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.462316 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c13b847-7063-4a20-ad97-48061bb6b8eb-kube-api-access-7zdln\" (UniqueName: \"kubernetes.io/projected/8c13b847-7063-4a20-ad97-48061bb6b8eb-kube-api-access-7zdln\") pod \"8c13b847-7063-4a20-ad97-48061bb6b8eb\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " Oct 30 18:18:52.462438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.462425 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-openshift-ca\") pod \"8c13b847-7063-4a20-ad97-48061bb6b8eb\" (UID: \"8c13b847-7063-4a20-ad97-48061bb6b8eb\") " Oct 30 18:18:52.464263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.464224 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-openshift-ca" pod "8c13b847-7063-4a20-ad97-48061bb6b8eb" (UID: "8c13b847-7063-4a20-ad97-48061bb6b8eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:52.464263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.464626 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-xks-ca" pod "8c13b847-7063-4a20-ad97-48061bb6b8eb" (UID: "8c13b847-7063-4a20-ad97-48061bb6b8eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:52.464263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.464936 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c13b847-7063-4a20-ad97-48061bb6b8eb-kube-api-access-7zdln" pod "8c13b847-7063-4a20-ad97-48061bb6b8eb" (UID: "8c13b847-7063-4a20-ad97-48061bb6b8eb"). InnerVolumeSpecName "kube-api-access-7zdln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:52.564761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.564723 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:52.564761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.564756 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zdln\" (UniqueName: \"kubernetes.io/projected/8c13b847-7063-4a20-ad97-48061bb6b8eb-kube-api-access-7zdln\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:52.564761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:52.564768 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c13b847-7063-4a20-ad97-48061bb6b8eb-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.147973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.147935 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" Oct 30 18:18:53.148153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.147976 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9" event={"ID":"8c13b847-7063-4a20-ad97-48061bb6b8eb","Type":"ContainerDied","Data":"72525b04b463e084d8b3867932614a7c6fdc1c938a48141460e52cfeaa44a331"} Oct 30 18:18:53.148153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.148045 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72525b04b463e084d8b3867932614a7c6fdc1c938a48141460e52cfeaa44a331" Oct 30 18:18:53.804433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.804408 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:18:53.880441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880407 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-data\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.880607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880450 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-certs\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.880607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880493 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-kube-api-access-wqhvq\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-kube-api-access-wqhvq\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.880607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880512 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-envoy\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.880800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880613 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-podinfo\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.880800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880729 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-data" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:53.880800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880744 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-credential-socket\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.881007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880809 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-certs" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:53.881007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880907 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-token\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-token\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.881007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880943 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-credential-socket" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:53.881007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.880999 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istiod-ca-cert\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.881221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881056 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-socket\") pod \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\" (UID: \"d65c4fcc-dfae-42df-9760-4c8feecd05a3\") " Oct 30 18:18:53.881670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881390 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istiod-ca-cert" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:53.881670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881636 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-socket" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:53.881875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881778 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.881875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881800 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.881875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881850 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.881875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881864 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.882086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.881878 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.884984 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.884958 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-podinfo" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:18:53.885101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.885015 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-envoy" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:53.885530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.885503 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-kube-api-access-wqhvq" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "kube-api-access-wqhvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:53.887033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.887008 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-token" pod "d65c4fcc-dfae-42df-9760-4c8feecd05a3" (UID: "d65c4fcc-dfae-42df-9760-4c8feecd05a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:53.983098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.983076 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.983098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.983098 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhvq\" (UniqueName: \"kubernetes.io/projected/d65c4fcc-dfae-42df-9760-4c8feecd05a3-kube-api-access-wqhvq\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.983239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.983108 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.983239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:53.983117 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d65c4fcc-dfae-42df-9760-4c8feecd05a3-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:18:54.128592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.128557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:54.157107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.157075 4582 generic.go:356] "Generic (PLEG): container finished" podID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerID="80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8" exitCode=0 Oct 30 18:18:54.157286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.157128 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" Oct 30 18:18:54.157286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.157151 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" event={"ID":"d65c4fcc-dfae-42df-9760-4c8feecd05a3","Type":"ContainerDied","Data":"80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8"} Oct 30 18:18:54.157286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.157200 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2" event={"ID":"d65c4fcc-dfae-42df-9760-4c8feecd05a3","Type":"ContainerDied","Data":"dac5b9f1ba9b6b8a06f17ba8fb41bc833aa07ed9f8633b26a188870a087e9246"} Oct 30 18:18:54.157286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.157223 4582 scope.go:122] "RemoveContainer" containerID="80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8" Oct 30 18:18:54.168526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.168509 4582 scope.go:122] "RemoveContainer" containerID="a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f" Oct 30 18:18:54.178305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.178287 4582 scope.go:122] "RemoveContainer" containerID="80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8" Oct 30 18:18:54.178559 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:18:54.178543 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8\": container with ID starting with 80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8 not found: ID does not exist" containerID="80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8" Oct 30 18:18:54.178620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.178570 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8"} err="failed to get container status \"80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8\": rpc error: code = NotFound desc = could not find container \"80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8\": container with ID starting with 80cdd5a0bef3f5678a29bb83ecae290f11aa6a4b44c5e47fe2c8b93fa18dd0c8 not found: ID does not exist" Oct 30 18:18:54.178620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.178586 4582 scope.go:122] "RemoveContainer" containerID="a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f" Oct 30 18:18:54.178807 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:18:54.178785 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f\": container with ID starting with a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f not found: ID does not exist" containerID="a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f" Oct 30 18:18:54.178881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.178814 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f"} err="failed to get container status \"a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f\": rpc error: code = NotFound desc = could not find container \"a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f\": container with ID starting with a784f91b789b9a13d29e8a9fe69a6ccff7c644a5f36a8e7d6101c1ae7c3cf45f not found: ID does not exist" Oct 30 18:18:54.213865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.213799 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2"] Oct 30 18:18:54.219245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:54.219213 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f15587-openshift-default-8554849b4f-t4hd2"] Oct 30 18:18:55.310845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:55.307384 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" path="/var/lib/kubelet/pods/d65c4fcc-dfae-42df-9760-4c8feecd05a3/volumes" Oct 30 18:18:56.150893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:56.150208 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:57.646293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:57.646251 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.429845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:58.427544 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.608921 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:58.608883 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.694748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:58.694643 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69"] Oct 30 18:18:58.704253 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:58.704221 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-gdp69"] Oct 30 18:18:58.790606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:58.790569 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:59.301176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:59.301139 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c31dfc-92b1-46f2-a6e5-cfb7e5839c89" path="/var/lib/kubelet/pods/97c31dfc-92b1-46f2-a6e5-cfb7e5839c89/volumes" Oct 30 18:18:59.775589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:59.775555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:59.869848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:18:59.869799 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:00.820483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:00.820448 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:02.248805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:02.248771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:02.863912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:02.863873 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:02.954262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:02.954222 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:03.113652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:03.113620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:03.243006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:03.242974 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf"] Oct 30 18:19:03.250096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:03.250070 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b33a49-g78tf"] Oct 30 18:19:03.303905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:03.303873 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f" path="/var/lib/kubelet/pods/80c5e3fb-e712-48c1-a8b0-ad2fadb45b5f/volumes" Oct 30 18:19:03.495578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:03.495487 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:03.732601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:03.732566 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:05.372725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:05.372686 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:06.226261 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:19:06.226218 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:19:06.647580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:06.647540 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:07.173868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:07.173700 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:07.469892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:07.469787 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:07.570365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:07.570320 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:09.136526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:09.136496 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:09.321431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:09.321403 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:09.419545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:09.419457 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:10.214459 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:10.214430 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:10.312946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:10.312907 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:10.934731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:10.934702 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.024638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:11.024590 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.180765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:11.180733 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.263703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:11.263630 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj"] Oct 30 18:19:11.264149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:11.263914 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" containerID="cri-o://b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229" gracePeriod=30 Oct 30 18:19:11.295284 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:11.295253 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.391754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:11.391721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.589868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:11.589758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.341377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:12.341328 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.535468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:12.535435 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.852783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:12.852736 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.985438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:12.985404 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.086263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.086227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.136684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.136651 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp"] Oct 30 18:19:13.137652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137629 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8c13b847-7063-4a20-ad97-48061bb6b8eb" containerName="revoke-keys" Oct 30 18:19:13.137652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137649 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c13b847-7063-4a20-ad97-48061bb6b8eb" containerName="revoke-keys" Oct 30 18:19:13.137883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137671 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" Oct 30 18:19:13.137883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137680 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" Oct 30 18:19:13.137883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137696 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" Oct 30 18:19:13.137883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137703 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" Oct 30 18:19:13.137883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137858 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c13b847-7063-4a20-ad97-48061bb6b8eb" containerName="revoke-keys" Oct 30 18:19:13.137883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137872 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" Oct 30 18:19:13.137883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.137882 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65c4fcc-dfae-42df-9760-4c8feecd05a3" containerName="istio-proxy" Oct 30 18:19:13.146078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.146057 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.163612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.162860 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f42594-openshift-default-dockercfg-dj4xj\"" Oct 30 18:19:13.168715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.168689 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp"] Oct 30 18:19:13.189532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.189499 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.195074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195051 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-credential-socket\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195127 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-envoy\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195220 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-certs\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195439 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195414 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-socket\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195487 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-data\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195522 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-podinfo\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195559 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2mv\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-kube-api-access-gn2mv\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195696 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istiod-ca-cert\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.195808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.195792 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-token\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297458 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-socket\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297530 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-data\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297558 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-podinfo\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297594 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2mv\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-kube-api-access-gn2mv\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297672 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istiod-ca-cert\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297758 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-token\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297895 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-credential-socket\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.297949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297936 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-socket\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.298111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.297968 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-envoy\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.298111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.298001 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-certs\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.298338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.298318 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-certs\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.298537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.298511 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-credential-socket\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.298694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.298529 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istiod-ca-cert\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.299339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.299314 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-data\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.300981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.300962 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-envoy\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.301547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.301528 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-podinfo\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.331921 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.331895 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2mv\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-kube-api-access-gn2mv\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.336207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.336183 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-token\") pod \"e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.467341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.467016 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:13.474601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.474568 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.772693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.772602 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.863840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.863785 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp"] Oct 30 18:19:13.865366 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:19:13.865332 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8a245c_4bcd_410b_be05_47e8e1abd4fb.slice/crio-cd9eea4a41ecd7b2e6b11d2988188998d83d7ad2ac6d55a023a4bf15029b03e0 WatchSource:0}: Error finding container cd9eea4a41ecd7b2e6b11d2988188998d83d7ad2ac6d55a023a4bf15029b03e0: Status 404 returned error can't find the container with id cd9eea4a41ecd7b2e6b11d2988188998d83d7ad2ac6d55a023a4bf15029b03e0 Oct 30 18:19:13.868325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.868290 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:13.868442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.868373 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:13.868442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.868421 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:13.889855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.889815 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.992790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:13.992751 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:14.336854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:14.336773 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:14.544861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:14.544809 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" event={"ID":"3f8a245c-4bcd-410b-be05-47e8e1abd4fb","Type":"ContainerStarted","Data":"f9d2738ae8c5fec7c14743a8a093e431b2977e245644370fd270bb2c238f483f"} Oct 30 18:19:14.545338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:14.544921 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" event={"ID":"3f8a245c-4bcd-410b-be05-47e8e1abd4fb","Type":"ContainerStarted","Data":"cd9eea4a41ecd7b2e6b11d2988188998d83d7ad2ac6d55a023a4bf15029b03e0"} Oct 30 18:19:14.603460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:14.603367 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podStartSLOduration=1.603354631 podStartE2EDuration="1.603354631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:14.592168839 +0000 UTC m=+1251.416563131" watchObservedRunningTime="2026-10-30 18:19:14.603354631 +0000 UTC m=+1251.427748922" Oct 30 18:19:15.468868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:15.468834 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:15.474070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:15.474046 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:15.556463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:15.556432 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:15.557540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:15.557518 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:15.853592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:15.853497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:16.550602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.550547 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:19:16.566888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.566859 4582 generic.go:356] "Generic (PLEG): container finished" podID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerID="b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229" exitCode=0 Oct 30 18:19:16.567296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.566906 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" Oct 30 18:19:16.567296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.566938 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" event={"ID":"732c9a0e-d8dc-4b8a-ab65-beae822b07da","Type":"ContainerDied","Data":"b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229"} Oct 30 18:19:16.567296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.567002 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj" event={"ID":"732c9a0e-d8dc-4b8a-ab65-beae822b07da","Type":"ContainerDied","Data":"85deb478d2be16f77f276d704af80a3ece9f95989e17aa704a01d0dea8a11dae"} Oct 30 18:19:16.567296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.567036 4582 scope.go:122] "RemoveContainer" containerID="b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229" Oct 30 18:19:16.577692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.577674 4582 scope.go:122] "RemoveContainer" containerID="107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1" Oct 30 18:19:16.587749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.587732 4582 scope.go:122] "RemoveContainer" containerID="b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229" Oct 30 18:19:16.588073 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:19:16.588053 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229\": container with ID starting with b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229 not found: ID does not exist" containerID="b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229" Oct 30 18:19:16.588157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.588079 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229"} err="failed to get container status \"b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229\": rpc error: code = NotFound desc = could not find container \"b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229\": container with ID starting with b6fb592186382d80c17a7b4f64e49b77cfd8555c14f2f1ea2312adc7aa884229 not found: ID does not exist" Oct 30 18:19:16.588157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.588100 4582 scope.go:122] "RemoveContainer" containerID="107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1" Oct 30 18:19:16.588381 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:19:16.588358 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1\": container with ID starting with 107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1 not found: ID does not exist" containerID="107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1" Oct 30 18:19:16.588431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.588392 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1"} err="failed to get container status \"107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1\": rpc error: code = NotFound desc = could not find container \"107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1\": container with ID starting with 107599c278a596c677e9a03498ae4a045dfbd69c73782a1593938ffc9603baa1 not found: ID does not exist" Oct 30 18:19:16.669681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.669653 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-token\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-token\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.669907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.669714 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-certs\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.669907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.669851 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-kube-api-access-8fl5c\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-kube-api-access-8fl5c\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.669907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.669889 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istiod-ca-cert\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.670100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.669994 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-envoy\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.670100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670076 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-certs" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.670214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670104 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-data\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.670267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670228 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istiod-ca-cert" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:16.670267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670243 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-podinfo\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.670366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670276 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-socket\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.670366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670313 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-credential-socket\") pod \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\" (UID: \"732c9a0e-d8dc-4b8a-ab65-beae822b07da\") " Oct 30 18:19:16.670470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670359 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-data" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.670612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670582 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-socket" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.670695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.670607 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-credential-socket" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.671213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.671194 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.671213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.671211 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.671395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.671221 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.671395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.671234 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.671395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.671248 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.672263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.672240 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-token" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:16.672436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.672412 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-envoy" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.672507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.672438 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-kube-api-access-8fl5c" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "kube-api-access-8fl5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:16.672543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.672521 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-podinfo" pod "732c9a0e-d8dc-4b8a-ab65-beae822b07da" (UID: "732c9a0e-d8dc-4b8a-ab65-beae822b07da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:16.772891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.772849 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fl5c\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-kube-api-access-8fl5c\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.772891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.772878 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.772891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.772888 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.772891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.772896 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732c9a0e-d8dc-4b8a-ab65-beae822b07da-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.915522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.915491 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj"] Oct 30 18:19:16.924611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:16.924586 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b33a49-openshift-default-5cf545bb67-kbrdj"] Oct 30 18:19:17.193734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:17.193638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.302083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:17.302046 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" path="/var/lib/kubelet/pods/732c9a0e-d8dc-4b8a-ab65-beae822b07da/volumes" Oct 30 18:19:17.318341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:17.318314 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.418155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:17.418120 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.532216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:17.532129 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.636794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:17.636746 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.791104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:17.790985 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:18.955896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:18.955798 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:19.057964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:19.057930 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:19.951333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:19.951295 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.659956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:20.659919 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.763792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:20.763763 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:21.029728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:21.029648 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk"] Oct 30 18:19:21.038451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:21.038427 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5da84a-tfzsk"] Oct 30 18:19:21.305242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:21.305140 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a9221d-6334-4fd5-ba87-0c50a64e78fe" path="/var/lib/kubelet/pods/67a9221d-6334-4fd5-ba87-0c50a64e78fe/volumes" Oct 30 18:19:21.844187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:21.844153 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:23.519307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:23.519257 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:24.994594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:24.994540 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:25.095664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:25.095631 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:25.246911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:25.246748 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:26.920635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:26.920602 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:28.648878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:28.648817 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:29.223653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:29.223622 4582 scope.go:122] "RemoveContainer" containerID="d61caefdfc0e9a5d7d2bbf56732484cc6be03f773f5e0d3ad9a6cdf2b021e7b8" Oct 30 18:19:29.233855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:29.233800 4582 scope.go:122] "RemoveContainer" containerID="2833bf406ff2a9b1d1ed4a1d9533e4712b99f780853accb87f74382fbd173de6" Oct 30 18:19:29.243274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:29.243251 4582 scope.go:122] "RemoveContainer" containerID="24ca5d59fb0db3771e6f1bffb45175a0183d74b361159a7b44654cf1b1a8dab2" Oct 30 18:19:29.906933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:29.906897 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:30.545502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:30.545466 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:30.837710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:30.837620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:30.878365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:30.878328 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9"] Oct 30 18:19:30.882587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:30.882564 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8dc43c-9trz9"] Oct 30 18:19:30.922389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:30.922361 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:31.301956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:31.301924 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c13b847-7063-4a20-ad97-48061bb6b8eb" path="/var/lib/kubelet/pods/8c13b847-7063-4a20-ad97-48061bb6b8eb/volumes" Oct 30 18:19:31.367107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:31.367076 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.808182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.808151 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.897482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.897444 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.990286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.990255 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t"] Oct 30 18:19:32.990924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.990909 4582 cpu_manager.go:414] "containerMap: removing container" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" Oct 30 18:19:32.990977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.990925 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" Oct 30 18:19:32.991063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.991054 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" Oct 30 18:19:32.993043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.993020 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.997173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:32.997156 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.009750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.009717 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:19:33.017015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.016989 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t"] Oct 30 18:19:33.033969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.033936 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:19:33.034202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.034070 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4jrp2\"" Oct 30 18:19:33.034202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.034104 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:19:33.145850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.145797 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.146004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.145910 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.146004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.145971 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47123ed0-00ec-47a3-88ad-c88861788157-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.146105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.146089 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.146157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.146143 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjh5\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-kube-api-access-rwjh5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.146231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.146209 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.146291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.146277 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.248234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.248199 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.248436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.248239 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.248436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.248261 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47123ed0-00ec-47a3-88ad-c88861788157-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.248436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.248337 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.248436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.248386 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjh5\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-kube-api-access-rwjh5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.248436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.248408 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.248729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.248457 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.249164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.249133 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.250726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.250700 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47123ed0-00ec-47a3-88ad-c88861788157-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.250878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.250864 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.250994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.250975 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.251075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.251055 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.264580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.264558 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.264673 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.264587 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjh5\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-kube-api-access-rwjh5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4k2t\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.315745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.315723 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:33.637705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.637674 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t"] Oct 30 18:19:33.640188 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:19:33.639903 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47123ed0_00ec_47a3_88ad_c88861788157.slice/crio-bf255da526a98f3be7da135bf78f312d6635ce47180179301d567a133c1b6295 WatchSource:0}: Error finding container bf255da526a98f3be7da135bf78f312d6635ce47180179301d567a133c1b6295: Status 404 returned error can't find the container with id bf255da526a98f3be7da135bf78f312d6635ce47180179301d567a133c1b6295 Oct 30 18:19:33.744993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:33.744957 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" event={"ID":"47123ed0-00ec-47a3-88ad-c88861788157","Type":"ContainerStarted","Data":"bf255da526a98f3be7da135bf78f312d6635ce47180179301d567a133c1b6295"} Oct 30 18:19:34.624369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:34.624302 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:34.724583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:34.724534 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:34.826117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:34.826086 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:35.028171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:35.028085 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt"] Oct 30 18:19:35.037380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:35.037335 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc061a6-4twpt"] Oct 30 18:19:35.301870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:35.301743 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c01f61-0115-452d-b93d-f8810966f95d" path="/var/lib/kubelet/pods/55c01f61-0115-452d-b93d-f8810966f95d/volumes" Oct 30 18:19:36.233011 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:19:36.232970 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:19:36.372903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:36.372866 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:36.672099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:36.672069 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:36.672191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:36.672133 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:36.776088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:36.776033 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" event={"ID":"47123ed0-00ec-47a3-88ad-c88861788157","Type":"ContainerStarted","Data":"7231aaa1d10f46124942763d36c99563f916925b7aba2364f6ab8ab2ad7b813a"} Oct 30 18:19:36.776221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:36.776127 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:36.836493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:36.836442 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" podStartSLOduration=1.808770055 podStartE2EDuration="4.836428756s" totalImagesPullingTime="3.027658701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:36.823989729 +0000 UTC m=+1273.648384040" watchObservedRunningTime="2026-10-30 18:19:36.836428756 +0000 UTC m=+1273.660823047" Oct 30 18:19:37.734787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:37.734752 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:37.785613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:37.785574 4582 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r4k2t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:19:37.785775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:37.785649 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" podUID="47123ed0-00ec-47a3-88ad-c88861788157" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:19:37.839521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:37.839483 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:38.065274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:38.065192 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7"] Oct 30 18:19:38.065480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:38.065438 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" containerID="cri-o://d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1" gracePeriod=30 Oct 30 18:19:38.085158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:38.085102 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:38.185495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:38.185454 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:39.861791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:39.861756 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:40.784392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:40.784365 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:19:40.856613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:40.856579 4582 patch_prober.go:29] interesting pod/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:19:40.856770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:40.856633 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:19:41.641078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:41.641035 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:43.341920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.341867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:43.361193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.361170 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:19:43.430333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430295 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-envoy\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.430542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430349 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-token\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-token\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.430542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430384 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-podinfo\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.430542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430418 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-kube-api-access-d7k86\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-kube-api-access-d7k86\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.430542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430448 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-certs\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.430931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430699 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cfbc4b9-c8ae-4011-869a-a86642c92644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfbc4b9-c8ae-4011-869a-a86642c92644-istiod-ca-cert\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.430931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430748 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-credential-socket\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.430931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.430847 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-socket\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.431175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.431037 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-data\") pod \"9cfbc4b9-c8ae-4011-869a-a86642c92644\" (UID: \"9cfbc4b9-c8ae-4011-869a-a86642c92644\") " Oct 30 18:19:43.431583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.431556 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-certs" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:43.432718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.432380 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cfbc4b9-c8ae-4011-869a-a86642c92644-istiod-ca-cert" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:43.432718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.432599 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-credential-socket" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:43.433326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.432760 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-socket" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:43.433326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.432946 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-data" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:43.433687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.433658 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-envoy" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:43.465195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.460492 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-podinfo" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:43.465195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.461435 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.478927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.465572 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-kube-api-access-d7k86" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "kube-api-access-d7k86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:43.570134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.569933 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfbc4b9-c8ae-4011-869a-a86642c92644-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.570134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.569975 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.570134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.569990 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.570134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.570003 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.570134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.570015 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.570134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.570027 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.570134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.570040 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k86\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-kube-api-access-d7k86\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.574041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.574010 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-token" pod "9cfbc4b9-c8ae-4011-869a-a86642c92644" (UID: "9cfbc4b9-c8ae-4011-869a-a86642c92644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:43.671427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.671392 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfbc4b9-c8ae-4011-869a-a86642c92644-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:19:43.841884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.841847 4582 generic.go:356] "Generic (PLEG): container finished" podID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerID="d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1" exitCode=0 Oct 30 18:19:43.842060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.841904 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" Oct 30 18:19:43.842060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.841916 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" event={"ID":"9cfbc4b9-c8ae-4011-869a-a86642c92644","Type":"ContainerDied","Data":"d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1"} Oct 30 18:19:43.842060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.841973 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7" event={"ID":"9cfbc4b9-c8ae-4011-869a-a86642c92644","Type":"ContainerDied","Data":"dcd7f998c1e5f1a902aa88e22303246874bb88a4b90baf0eb3d8228b1f7e1c77"} Oct 30 18:19:43.842060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.841995 4582 scope.go:122] "RemoveContainer" containerID="d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1" Oct 30 18:19:43.853029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.853010 4582 scope.go:122] "RemoveContainer" containerID="baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae" Oct 30 18:19:43.862887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.862867 4582 scope.go:122] "RemoveContainer" containerID="d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1" Oct 30 18:19:43.863161 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:19:43.863138 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1\": container with ID starting with d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1 not found: ID does not exist" containerID="d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1" Oct 30 18:19:43.863228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.863171 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1"} err="failed to get container status \"d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1\": rpc error: code = NotFound desc = could not find container \"d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1\": container with ID starting with d34c5a43986b36869b3821ce899b90785f466c33f04a6aedd3acbaf15679a5c1 not found: ID does not exist" Oct 30 18:19:43.863228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.863191 4582 scope.go:122] "RemoveContainer" containerID="baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae" Oct 30 18:19:43.863444 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:19:43.863426 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae\": container with ID starting with baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae not found: ID does not exist" containerID="baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae" Oct 30 18:19:43.863496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.863448 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae"} err="failed to get container status \"baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae\": rpc error: code = NotFound desc = could not find container \"baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae\": container with ID starting with baa19961ff9112e674f03252cb583b295e72ecfbb511ff449155809b0c7a00ae not found: ID does not exist" Oct 30 18:19:43.891659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.891628 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7"] Oct 30 18:19:43.900753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:43.900727 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8dc43c-openshift-default-5cb789c645-bz9k7"] Oct 30 18:19:45.056948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:45.056911 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw"] Oct 30 18:19:45.081216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:45.077114 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:45.081216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:45.081113 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb289cc0-9hsnw"] Oct 30 18:19:45.301133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:45.301101 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" path="/var/lib/kubelet/pods/9cfbc4b9-c8ae-4011-869a-a86642c92644/volumes" Oct 30 18:19:45.301639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:45.301625 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd67b9c-9b01-49fd-9958-8c1b165a6643" path="/var/lib/kubelet/pods/bfd67b9c-9b01-49fd-9958-8c1b165a6643/volumes" Oct 30 18:19:45.460766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:45.460732 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:47.472847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:47.472298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:49.328495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:49.328422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:49.430446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:49.430413 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:49.800369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:49.800337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:50.271595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:50.271559 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:51.027836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:51.027791 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz"] Oct 30 18:19:51.036234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:51.036207 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af32be1-ws6bz"] Oct 30 18:19:51.305411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:51.305332 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34388c06-6a2e-4b53-b2e9-548a99e2535f" path="/var/lib/kubelet/pods/34388c06-6a2e-4b53-b2e9-548a99e2535f/volumes" Oct 30 18:19:51.445315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:51.445282 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:53.268311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:53.268266 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:54.021154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.021127 4582 generic.go:356] "Generic (PLEG): container finished" podID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerID="f9d2738ae8c5fec7c14743a8a093e431b2977e245644370fd270bb2c238f483f" exitCode=0 Oct 30 18:19:54.021308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.021200 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" event={"ID":"3f8a245c-4bcd-410b-be05-47e8e1abd4fb","Type":"ContainerDied","Data":"f9d2738ae8c5fec7c14743a8a093e431b2977e245644370fd270bb2c238f483f"} Oct 30 18:19:54.021615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.021600 4582 scope.go:122] "RemoveContainer" containerID="f9d2738ae8c5fec7c14743a8a093e431b2977e245644370fd270bb2c238f483f" Oct 30 18:19:54.023049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.023024 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:54.023119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.023088 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:54.023158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.023122 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:19:54.079461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.079435 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq"] Oct 30 18:19:54.094786 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.094754 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-329b5113-9bdhq"] Oct 30 18:19:54.995673 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:54.995631 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:55.032464 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:55.032428 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" event={"ID":"3f8a245c-4bcd-410b-be05-47e8e1abd4fb","Type":"ContainerStarted","Data":"96ccb13fb4e79ce42135b907ed1811660583cc2d61785f0e329a505a3cc632d1"} Oct 30 18:19:55.300655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:55.300579 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36f3041-677f-4970-b8bb-a3f5bc19f167" path="/var/lib/kubelet/pods/d36f3041-677f-4970-b8bb-a3f5bc19f167/volumes" Oct 30 18:19:55.468177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:55.468142 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:19:55.468421 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:55.468396 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:19:55.468493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:55.468470 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:19:56.468245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:56.468209 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:19:56.468661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:56.468265 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:19:56.868785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:56.868753 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:57.468390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:57.468349 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:19:57.468935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:57.468422 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:19:57.874486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:57.874318 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:57.979028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:57.978984 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:58.027508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:58.027468 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl"] Oct 30 18:19:58.038061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:58.038029 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf91da-bb5vl"] Oct 30 18:19:58.471237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:58.471200 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:19:58.472264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:58.471862 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:19:58.669519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:58.669485 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:59.301576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:59.301547 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f090e2d-6193-4bd2-975c-8468554fbaa4" path="/var/lib/kubelet/pods/4f090e2d-6193-4bd2-975c-8468554fbaa4/volumes" Oct 30 18:19:59.468709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:59.468672 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:19:59.468906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:59.468739 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:19:59.982151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:19:59.982103 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:00.341002 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:00.340914 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:00.472639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:00.472611 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:20:00.473082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:00.473061 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:20:00.473710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:00.473695 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:20:01.096608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:01.096573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:01.738523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:01.738489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.107210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.107131 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.150457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.150419 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb"] Oct 30 18:20:02.151320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151296 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" Oct 30 18:20:02.151320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151316 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" Oct 30 18:20:02.151531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151342 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" Oct 30 18:20:02.151531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151347 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" Oct 30 18:20:02.151531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151383 4582 cpu_manager.go:414] "containerMap: removing container" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" Oct 30 18:20:02.151531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151388 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" Oct 30 18:20:02.151531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151486 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c9a0e-d8dc-4b8a-ab65-beae822b07da" containerName="istio-proxy" Oct 30 18:20:02.151531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151494 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" Oct 30 18:20:02.151531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.151504 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cfbc4b9-c8ae-4011-869a-a86642c92644" containerName="istio-proxy" Oct 30 18:20:02.157368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.157346 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.171657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.171631 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-de14f2-openshift-default-dockercfg-v4w96\"" Oct 30 18:20:02.176017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.175991 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb"] Oct 30 18:20:02.205358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.205323 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.335924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.335879 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0fff5-1c1b-4acd-b897-36983b924d27-istiod-ca-cert\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.335933 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-token\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.336108 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-certs\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.336160 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-envoy\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.336320 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-podinfo\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.336494 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-data\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.336582 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-socket\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.336616 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrjm\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-kube-api-access-6wrjm\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.336720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.336687 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-credential-socket\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438160 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-data\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438254 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-socket\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438284 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrjm\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-kube-api-access-6wrjm\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438327 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-credential-socket\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438419 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0fff5-1c1b-4acd-b897-36983b924d27-istiod-ca-cert\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438443 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-token\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438563 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-certs\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438615 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-envoy\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.438752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.438730 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-podinfo\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.444702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.441259 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-socket\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.444702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.442267 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0fff5-1c1b-4acd-b897-36983b924d27-istiod-ca-cert\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.444702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.442428 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-certs\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.444702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.442979 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-data\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.444702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.443783 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-credential-socket\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.456396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.450911 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.456396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.455107 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-envoy\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.462702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.462648 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrjm\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-kube-api-access-6wrjm\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.465528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.465049 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-token\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.467318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.467293 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-podinfo\") pod \"e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.483448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.483422 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:02.810657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.810627 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb"] Oct 30 18:20:02.813444 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:20:02.813413 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e0fff5_1c1b_4acd_b897_36983b924d27.slice/crio-cfb850d3d0c7a009e05b791aec272399555457b1d80e119e5a7388238df7bb84 WatchSource:0}: Error finding container cfb850d3d0c7a009e05b791aec272399555457b1d80e119e5a7388238df7bb84: Status 404 returned error can't find the container with id cfb850d3d0c7a009e05b791aec272399555457b1d80e119e5a7388238df7bb84 Oct 30 18:20:02.816840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.816709 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:20:02.816840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.816796 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:20:02.817017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:02.816889 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:20:03.129363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:03.129330 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" event={"ID":"88e0fff5-1c1b-4acd-b897-36983b924d27","Type":"ContainerStarted","Data":"bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a"} Oct 30 18:20:03.129911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:03.129381 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" event={"ID":"88e0fff5-1c1b-4acd-b897-36983b924d27","Type":"ContainerStarted","Data":"cfb850d3d0c7a009e05b791aec272399555457b1d80e119e5a7388238df7bb84"} Oct 30 18:20:03.164724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:03.164672 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" podStartSLOduration=1.1646580069999999 podStartE2EDuration="1.164658007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:03.163925033 +0000 UTC m=+1299.988319325" watchObservedRunningTime="2026-10-30 18:20:03.164658007 +0000 UTC m=+1299.989052299" Oct 30 18:20:03.214059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:03.214024 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:03.318908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:03.318869 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:03.500916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:03.492923 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:03.500916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:03.498459 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:04.137683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:04.137653 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:04.138616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:04.138599 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:04.503010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:04.502911 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:04.918170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:04.918141 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:05.017899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:05.017816 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:05.890568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:05.890532 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:05.993121 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:05.993087 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:06.109882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.109840 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp"] Oct 30 18:20:06.115777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.115753 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.132209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.132182 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp"] Oct 30 18:20:06.133390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.133371 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:20:06.133390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.133371 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:20:06.220587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.220490 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:20:06.220587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.220542 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:20:06.226857 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:20:06.226805 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:20:06.303466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.303435 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.303661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.303611 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvx8h\" (UniqueName: \"kubernetes.io/projected/dddb8050-8e9e-469c-b0b5-142177a8544f-kube-api-access-jvx8h\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.303724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.303662 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.404967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.404933 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.405149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.405024 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvx8h\" (UniqueName: \"kubernetes.io/projected/dddb8050-8e9e-469c-b0b5-142177a8544f-kube-api-access-jvx8h\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.405149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.405071 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.405720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.405696 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.407440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.407409 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.424558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.424507 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvx8h\" (UniqueName: \"kubernetes.io/projected/dddb8050-8e9e-469c-b0b5-142177a8544f-kube-api-access-jvx8h\") pod \"maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.431233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.431204 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:06.635527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.635486 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:06.748070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:06.743946 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp"] Oct 30 18:20:07.176404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:07.176353 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" event={"ID":"dddb8050-8e9e-469c-b0b5-142177a8544f","Type":"ContainerStarted","Data":"67d4dacc621e42f34074c3e9fcbdb83ca89503cdf60abc3d271b6a9c1c6ff399"} Oct 30 18:20:07.176913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:07.176441 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" event={"ID":"dddb8050-8e9e-469c-b0b5-142177a8544f","Type":"ContainerStarted","Data":"adc808c5f5ff5eb8f60422472be9490c222a710ebb9c3e05e29414915d780405"} Oct 30 18:20:07.214868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:07.214804 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" podStartSLOduration=1.214790403 podStartE2EDuration="1.214790403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:07.209224757 +0000 UTC m=+1304.033619049" watchObservedRunningTime="2026-10-30 18:20:07.214790403 +0000 UTC m=+1304.039184724" Oct 30 18:20:07.604283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:07.604248 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:07.700155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:07.700125 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.020888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:08.006862 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.112008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:08.111973 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.199148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:08.199115 4582 generic.go:356] "Generic (PLEG): container finished" podID="dddb8050-8e9e-469c-b0b5-142177a8544f" containerID="67d4dacc621e42f34074c3e9fcbdb83ca89503cdf60abc3d271b6a9c1c6ff399" exitCode=0 Oct 30 18:20:08.200222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:08.199203 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" event={"ID":"dddb8050-8e9e-469c-b0b5-142177a8544f","Type":"ContainerDied","Data":"67d4dacc621e42f34074c3e9fcbdb83ca89503cdf60abc3d271b6a9c1c6ff399"} Oct 30 18:20:08.226751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:08.226717 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.328936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:08.328812 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.811112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:08.811057 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:09.047551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.047521 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp"] Oct 30 18:20:09.055279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.055250 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb96a6a-kkxvp"] Oct 30 18:20:09.308033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.308000 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481c1c02-3b8c-4a58-a62a-ab5a7579ffe1" path="/var/lib/kubelet/pods/481c1c02-3b8c-4a58-a62a-ab5a7579ffe1/volumes" Oct 30 18:20:09.401083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.401057 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:09.462125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.462032 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddb8050-8e9e-469c-b0b5-142177a8544f-kube-api-access-jvx8h\" (UniqueName: \"kubernetes.io/projected/dddb8050-8e9e-469c-b0b5-142177a8544f-kube-api-access-jvx8h\") pod \"dddb8050-8e9e-469c-b0b5-142177a8544f\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " Oct 30 18:20:09.462295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.462140 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-xks-ca\") pod \"dddb8050-8e9e-469c-b0b5-142177a8544f\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " Oct 30 18:20:09.462379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.462363 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-openshift-ca\") pod \"dddb8050-8e9e-469c-b0b5-142177a8544f\" (UID: \"dddb8050-8e9e-469c-b0b5-142177a8544f\") " Oct 30 18:20:09.464428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.463738 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-openshift-ca" pod "dddb8050-8e9e-469c-b0b5-142177a8544f" (UID: "dddb8050-8e9e-469c-b0b5-142177a8544f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:09.467278 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.467227 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-xks-ca" pod "dddb8050-8e9e-469c-b0b5-142177a8544f" (UID: "dddb8050-8e9e-469c-b0b5-142177a8544f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:09.468280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.468252 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddb8050-8e9e-469c-b0b5-142177a8544f-kube-api-access-jvx8h" pod "dddb8050-8e9e-469c-b0b5-142177a8544f" (UID: "dddb8050-8e9e-469c-b0b5-142177a8544f"). InnerVolumeSpecName "kube-api-access-jvx8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:09.564927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.564899 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:09.564927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.564923 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvx8h\" (UniqueName: \"kubernetes.io/projected/dddb8050-8e9e-469c-b0b5-142177a8544f-kube-api-access-jvx8h\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:09.564927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:09.564935 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddb8050-8e9e-469c-b0b5-142177a8544f-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:10.217599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:10.217570 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" event={"ID":"dddb8050-8e9e-469c-b0b5-142177a8544f","Type":"ContainerDied","Data":"adc808c5f5ff5eb8f60422472be9490c222a710ebb9c3e05e29414915d780405"} Oct 30 18:20:10.217599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:10.217589 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp" Oct 30 18:20:10.217784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:10.217616 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc808c5f5ff5eb8f60422472be9490c222a710ebb9c3e05e29414915d780405" Oct 30 18:20:10.884587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:10.884554 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:11.030792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:11.030748 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w"] Oct 30 18:20:11.035764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:11.035739 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba370bee-lcz8w"] Oct 30 18:20:11.302418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:11.302347 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5203f7c-de3e-4d11-ad89-b54ec1f2a21e" path="/var/lib/kubelet/pods/c5203f7c-de3e-4d11-ad89-b54ec1f2a21e/volumes" Oct 30 18:20:12.122153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.122114 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:12.302456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.301136 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:12.712779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.712744 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h"] Oct 30 18:20:12.713684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.713654 4582 cpu_manager.go:414] "containerMap: removing container" podUID="dddb8050-8e9e-469c-b0b5-142177a8544f" containerName="revoke-keys" Oct 30 18:20:12.713684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.713683 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddb8050-8e9e-469c-b0b5-142177a8544f" containerName="revoke-keys" Oct 30 18:20:12.713928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.713885 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddb8050-8e9e-469c-b0b5-142177a8544f" containerName="revoke-keys" Oct 30 18:20:12.727090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.727063 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:12.741738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.739981 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h"] Oct 30 18:20:12.746240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.746197 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f42594\"" Oct 30 18:20:12.787127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.787093 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f42594-dockercfg-dlr56\"" Oct 30 18:20:12.819842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.819455 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:12.822707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.819965 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmrd\" (UniqueName: \"kubernetes.io/projected/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-kube-api-access-vsmrd\") pod \"payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:12.871913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.871861 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84"] Oct 30 18:20:12.882744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.882714 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:12.896871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.896815 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84"] Oct 30 18:20:12.921397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.921367 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmrd\" (UniqueName: \"kubernetes.io/projected/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-kube-api-access-vsmrd\") pod \"payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:12.921565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.921463 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:12.921565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.921553 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0467aa-1432-4f65-92d9-85331482a312-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:12.921800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.921773 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4j4\" (UniqueName: \"kubernetes.io/projected/4d0467aa-1432-4f65-92d9-85331482a312-kube-api-access-ms4j4\") pod \"payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:12.922221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.922200 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:12.937902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:12.937876 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmrd\" (UniqueName: \"kubernetes.io/projected/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-kube-api-access-vsmrd\") pod \"payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:13.023444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.023366 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0467aa-1432-4f65-92d9-85331482a312-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:13.023585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.023533 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4j4\" (UniqueName: \"kubernetes.io/projected/4d0467aa-1432-4f65-92d9-85331482a312-kube-api-access-ms4j4\") pod \"payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:13.024085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.024065 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0467aa-1432-4f65-92d9-85331482a312-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:13.039957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.039925 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4j4\" (UniqueName: \"kubernetes.io/projected/4d0467aa-1432-4f65-92d9-85331482a312-kube-api-access-ms4j4\") pod \"payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:13.061695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.061673 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:13.218639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.218605 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:13.364684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.363726 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h"] Oct 30 18:20:13.364684 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:20:13.364211 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cbd5d9_3f82_4d7f_90a2_04d3b9cd1c72.slice/crio-111cb07b55dbe55f3ebc0b7cbcd70fd97139d19a75e4b61b3228489c7a874f24 WatchSource:0}: Error finding container 111cb07b55dbe55f3ebc0b7cbcd70fd97139d19a75e4b61b3228489c7a874f24: Status 404 returned error can't find the container with id 111cb07b55dbe55f3ebc0b7cbcd70fd97139d19a75e4b61b3228489c7a874f24 Oct 30 18:20:13.387112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.387082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:13.527534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.527338 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84"] Oct 30 18:20:13.528961 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:20:13.528793 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0467aa_1432_4f65_92d9_85331482a312.slice/crio-4e3d252d6b03b03331ace06e2f3feef78a5e41e0d70013c35ee23c122e4b4cdd WatchSource:0}: Error finding container 4e3d252d6b03b03331ace06e2f3feef78a5e41e0d70013c35ee23c122e4b4cdd: Status 404 returned error can't find the container with id 4e3d252d6b03b03331ace06e2f3feef78a5e41e0d70013c35ee23c122e4b4cdd Oct 30 18:20:13.871157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:13.871127 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:14.300178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:14.300064 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" event={"ID":"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72","Type":"ContainerStarted","Data":"111cb07b55dbe55f3ebc0b7cbcd70fd97139d19a75e4b61b3228489c7a874f24"} Oct 30 18:20:14.301736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:14.301700 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" event={"ID":"4d0467aa-1432-4f65-92d9-85331482a312","Type":"ContainerStarted","Data":"4e3d252d6b03b03331ace06e2f3feef78a5e41e0d70013c35ee23c122e4b4cdd"} Oct 30 18:20:14.519355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:14.519320 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:14.728106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:14.721340 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:15.317089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:15.317048 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" event={"ID":"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72","Type":"ContainerStarted","Data":"744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5"} Oct 30 18:20:15.317585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:15.317115 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:15.318763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:15.318731 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" event={"ID":"4d0467aa-1432-4f65-92d9-85331482a312","Type":"ContainerStarted","Data":"db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68"} Oct 30 18:20:15.318937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:15.318832 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:15.350670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:15.350607 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" podStartSLOduration=1.995580843 podStartE2EDuration="3.35058944s" totalImagesPullingTime="1.355008597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:15.349537718 +0000 UTC m=+1312.173932012" watchObservedRunningTime="2026-10-30 18:20:15.35058944 +0000 UTC m=+1312.174983733" Oct 30 18:20:15.385680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:15.385625 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" podStartSLOduration=2.195143096 podStartE2EDuration="3.385607348s" totalImagesPullingTime="1.190464252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:15.382424764 +0000 UTC m=+1312.206819056" watchObservedRunningTime="2026-10-30 18:20:15.385607348 +0000 UTC m=+1312.210001640" Oct 30 18:20:15.909455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:15.909420 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:16.705874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:16.704802 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:16.804176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:16.804143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:17.975668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:17.975637 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:19.746923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:19.746874 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:21.740485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:21.740452 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:23.495158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:23.495126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:23.595590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:23.595555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:24.029295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:24.029256 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs"] Oct 30 18:20:24.041189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:24.041160 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8"] Oct 30 18:20:24.046301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:24.046278 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:24.046639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:24.046620 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7263f035-h9nvs"] Oct 30 18:20:24.050161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:24.050144 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa2a1ed-t56h8"] Oct 30 18:20:25.266065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:25.266027 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:25.301683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:25.301653 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf8c547-3c30-4a28-b85d-0496605b81f7" path="/var/lib/kubelet/pods/3cf8c547-3c30-4a28-b85d-0496605b81f7/volumes" Oct 30 18:20:25.302158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:25.302144 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502a4996-fc3c-48a6-a77f-b660e8fc9cb9" path="/var/lib/kubelet/pods/502a4996-fc3c-48a6-a77f-b660e8fc9cb9/volumes" Oct 30 18:20:26.328906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:26.328878 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:20:26.329300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:26.328946 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:20:27.160971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:27.160936 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:27.240871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:27.240830 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:27.340647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:27.340615 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:29.035170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:29.035125 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:29.297164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:29.297091 4582 scope.go:122] "RemoveContainer" containerID="bd8c12d196e8034d7eae9316ea541ab6966ef00a9b815677439aa5e9ea073b12" Oct 30 18:20:29.481298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:29.481267 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:29.959627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:29.959578 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:30.392667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:30.392634 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:30.838752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:30.838655 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:31.509150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:31.509068 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:31.641366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:31.641318 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:31.806278 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:31.806181 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp"] Oct 30 18:20:31.810613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:31.810583 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0c6613ea-x4kjp"] Oct 30 18:20:32.214397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:32.214356 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:32.323021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:32.322971 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:33.301894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:33.301862 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddb8050-8e9e-469c-b0b5-142177a8544f" path="/var/lib/kubelet/pods/dddb8050-8e9e-469c-b0b5-142177a8544f/volumes" Oct 30 18:20:33.510783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:33.510746 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:33.617322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:33.617234 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:34.279930 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:34.279899 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:35.245976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:35.245943 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:35.343368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:35.343334 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:36.228382 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:20:36.228345 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:20:36.687240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:36.687205 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:37.033267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:37.033193 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw"] Oct 30 18:20:37.039858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:37.039834 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada9444e-8mxlw"] Oct 30 18:20:37.241678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:37.241646 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg"] Oct 30 18:20:37.301651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:37.301572 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:37.302640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:37.302618 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e65248-8fee-4bd4-99e4-4374a6c49169" path="/var/lib/kubelet/pods/23e65248-8fee-4bd4-99e4-4374a6c49169/volumes" Oct 30 18:20:38.845541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:38.845493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:39.032831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:39.032784 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs"] Oct 30 18:20:39.039791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:39.039766 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c60ef7-x27cs"] Oct 30 18:20:39.301536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:39.301506 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcdd1c2-6ccb-44e6-b568-797a68a9ef31" path="/var/lib/kubelet/pods/7dcdd1c2-6ccb-44e6-b568-797a68a9ef31/volumes" Oct 30 18:20:40.489264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:40.489229 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.027851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:42.027790 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:43.029423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:43.029394 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h"] Oct 30 18:20:43.037195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:43.037171 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d1f1c5-bg67h"] Oct 30 18:20:43.301215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:43.301138 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5" path="/var/lib/kubelet/pods/7f0d4529-4f7f-4a2d-b3f5-4c6d8f83e8c5/volumes" Oct 30 18:20:44.030401 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:44.030364 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx"] Oct 30 18:20:44.038104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:44.038073 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79aa899-l7mqx"] Oct 30 18:20:44.332964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:44.332871 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:44.756354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:44.756310 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:45.234602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:45.234567 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:45.301602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:45.301570 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d804c6bc-8b68-4a54-9cf4-4b435e188ab1" path="/var/lib/kubelet/pods/d804c6bc-8b68-4a54-9cf4-4b435e188ab1/volumes" Oct 30 18:20:45.332113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:45.332081 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:46.335447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:46.335413 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:48.039177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:48.039148 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf"] Oct 30 18:20:48.053585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:48.053544 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f6db4d-lgtlf"] Oct 30 18:20:48.057177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:48.057152 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx"] Oct 30 18:20:48.063102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:48.062988 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76531443-8zbpx"] Oct 30 18:20:48.135616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:48.135569 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:48.862351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:48.862301 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:48.964467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:48.964436 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:49.302451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:49.302410 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8955a515-94c1-42b3-b2c5-bbdfbc268825" path="/var/lib/kubelet/pods/8955a515-94c1-42b3-b2c5-bbdfbc268825/volumes" Oct 30 18:20:49.303307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:49.303282 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e92d9aa-b84a-461f-882d-b8fe80bc16e0" path="/var/lib/kubelet/pods/9e92d9aa-b84a-461f-882d-b8fe80bc16e0/volumes" Oct 30 18:20:49.736763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:49.736729 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:49.968223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:49.968191 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:50.714003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:50.713945 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:51.036204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:51.036122 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf"] Oct 30 18:20:51.048509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:51.048477 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02440f36-796nf"] Oct 30 18:20:51.303966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:51.303870 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e570823-83c1-457b-a606-834516c1fe1c" path="/var/lib/kubelet/pods/3e570823-83c1-457b-a606-834516c1fe1c/volumes" Oct 30 18:20:51.477213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:51.477177 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:51.574711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:51.574628 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:52.267549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.267503 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" podUID="9b802f37-fda6-4097-b93a-c5a08834f42c" containerName="main" containerID="cri-o://3e3516220afd36753e34a46bb5a454d9db294b97ca7e0f3999e9eff242b9b6b1" gracePeriod=45 Oct 30 18:20:52.658514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.658485 4582 generic.go:356] "Generic (PLEG): container finished" podID="9b802f37-fda6-4097-b93a-c5a08834f42c" containerID="3e3516220afd36753e34a46bb5a454d9db294b97ca7e0f3999e9eff242b9b6b1" exitCode=0 Oct 30 18:20:52.658661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.658529 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" event={"ID":"9b802f37-fda6-4097-b93a-c5a08834f42c","Type":"ContainerDied","Data":"3e3516220afd36753e34a46bb5a454d9db294b97ca7e0f3999e9eff242b9b6b1"} Oct 30 18:20:52.658661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.658559 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" event={"ID":"9b802f37-fda6-4097-b93a-c5a08834f42c","Type":"ContainerDied","Data":"ba82c30da14b29056f95a06d5d3e54af77e7ca7c2e995931234ac5e8e76624c7"} Oct 30 18:20:52.658661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.658571 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba82c30da14b29056f95a06d5d3e54af77e7ca7c2e995931234ac5e8e76624c7" Oct 30 18:20:52.660773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.660752 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:20:52.707844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.706162 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b802f37-fda6-4097-b93a-c5a08834f42c-kube-api-access-76vkm\" (UniqueName: \"kubernetes.io/projected/9b802f37-fda6-4097-b93a-c5a08834f42c-kube-api-access-76vkm\") pod \"9b802f37-fda6-4097-b93a-c5a08834f42c\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " Oct 30 18:20:52.707844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.706221 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-tmp-dir\") pod \"9b802f37-fda6-4097-b93a-c5a08834f42c\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " Oct 30 18:20:52.707844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.706480 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-home\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-home\") pod \"9b802f37-fda6-4097-b93a-c5a08834f42c\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " Oct 30 18:20:52.707844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.706536 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-model-cache\") pod \"9b802f37-fda6-4097-b93a-c5a08834f42c\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " Oct 30 18:20:52.707844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.706642 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b802f37-fda6-4097-b93a-c5a08834f42c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b802f37-fda6-4097-b93a-c5a08834f42c-tls-certs\") pod \"9b802f37-fda6-4097-b93a-c5a08834f42c\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " Oct 30 18:20:52.707844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.706674 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-dshm\") pod \"9b802f37-fda6-4097-b93a-c5a08834f42c\" (UID: \"9b802f37-fda6-4097-b93a-c5a08834f42c\") " Oct 30 18:20:52.709007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.708699 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-model-cache" pod "9b802f37-fda6-4097-b93a-c5a08834f42c" (UID: "9b802f37-fda6-4097-b93a-c5a08834f42c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:52.709426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.709405 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-tmp-dir" pod "9b802f37-fda6-4097-b93a-c5a08834f42c" (UID: "9b802f37-fda6-4097-b93a-c5a08834f42c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:52.709966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.709944 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-home" pod "9b802f37-fda6-4097-b93a-c5a08834f42c" (UID: "9b802f37-fda6-4097-b93a-c5a08834f42c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:52.714378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.714355 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b802f37-fda6-4097-b93a-c5a08834f42c-kube-api-access-76vkm" pod "9b802f37-fda6-4097-b93a-c5a08834f42c" (UID: "9b802f37-fda6-4097-b93a-c5a08834f42c"). InnerVolumeSpecName "kube-api-access-76vkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:52.714922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.714896 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-dshm" pod "9b802f37-fda6-4097-b93a-c5a08834f42c" (UID: "9b802f37-fda6-4097-b93a-c5a08834f42c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:52.715177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.715154 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:52.719407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.719371 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b802f37-fda6-4097-b93a-c5a08834f42c-tls-certs" pod "9b802f37-fda6-4097-b93a-c5a08834f42c" (UID: "9b802f37-fda6-4097-b93a-c5a08834f42c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:52.810025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.809996 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vkm\" (UniqueName: \"kubernetes.io/projected/9b802f37-fda6-4097-b93a-c5a08834f42c-kube-api-access-76vkm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:52.810025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.810022 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:52.810230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.810037 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:52.810230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.810050 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:52.810230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.810061 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b802f37-fda6-4097-b93a-c5a08834f42c-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:52.810230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:52.810070 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b802f37-fda6-4097-b93a-c5a08834f42c-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:20:53.627541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:53.627503 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" podUID="9b802f37-fda6-4097-b93a-c5a08834f42c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:53.666289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:53.666262 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg" Oct 30 18:20:53.719097 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:53.719051 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg"] Oct 30 18:20:53.726046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:53.726017 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0c6613ea/e2e-lifecycle-model-0c6613ea-kserve-78df4b7b8f-mlmvg"] Oct 30 18:20:54.605422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.605316 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:54.676315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.676281 4582 generic.go:356] "Generic (PLEG): container finished" podID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerID="bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a" exitCode=0 Oct 30 18:20:54.676710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.676359 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" event={"ID":"88e0fff5-1c1b-4acd-b897-36983b924d27","Type":"ContainerDied","Data":"bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a"} Oct 30 18:20:54.676918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.676904 4582 scope.go:122] "RemoveContainer" containerID="bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a" Oct 30 18:20:54.678611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.678587 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:20:54.678686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.678656 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:20:54.678724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.678688 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:20:54.706484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.706446 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:54.896595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:54.896565 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:55.301815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:55.301737 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b802f37-fda6-4097-b93a-c5a08834f42c" path="/var/lib/kubelet/pods/9b802f37-fda6-4097-b93a-c5a08834f42c/volumes" Oct 30 18:20:55.705129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:55.705092 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" event={"ID":"88e0fff5-1c1b-4acd-b897-36983b924d27","Type":"ContainerStarted","Data":"ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214"} Oct 30 18:20:56.181049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:56.181017 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:56.423963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:56.423853 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:56.484686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:56.484572 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:20:56.484910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:56.484874 4582 patch_prober.go:29] interesting pod/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:20:56.484989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:56.484939 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 18:20:57.081029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.080986 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh"] Oct 30 18:20:57.110431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.110305 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9b802f37-fda6-4097-b93a-c5a08834f42c" containerName="main" Oct 30 18:20:57.110431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.110348 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b802f37-fda6-4097-b93a-c5a08834f42c" containerName="main" Oct 30 18:20:57.116481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.115926 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b802f37-fda6-4097-b93a-c5a08834f42c" containerName="main" Oct 30 18:20:57.135610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.135553 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh"] Oct 30 18:20:57.135768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.135713 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.167842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.165688 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:20:57.167842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.165982 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:20:57.175767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.175747 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:20:57.176140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.175781 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:20:57.326170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.326135 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47ls\" (UniqueName: \"kubernetes.io/projected/063e6cb4-e1df-435b-a1ec-fad68c9341db-kube-api-access-n47ls\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.326340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.326179 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.326340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.326295 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.427747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.427708 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47ls\" (UniqueName: \"kubernetes.io/projected/063e6cb4-e1df-435b-a1ec-fad68c9341db-kube-api-access-n47ls\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.427954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.427771 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.427954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.427888 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.428480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.428457 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.430161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.430141 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.469467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.469436 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47ls\" (UniqueName: \"kubernetes.io/projected/063e6cb4-e1df-435b-a1ec-fad68c9341db-kube-api-access-n47ls\") pod \"maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.480347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.480319 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:20:57.484857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.484813 4582 patch_prober.go:29] interesting pod/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:20:57.485005 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.484898 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 18:20:57.783582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:57.783551 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh"] Oct 30 18:20:57.785324 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:20:57.785295 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063e6cb4_e1df_435b_a1ec_fad68c9341db.slice/crio-e93991babeee025b933ee86fc873fb8f6ffa0cf5ef933b07c2adb3292eb29d07 WatchSource:0}: Error finding container e93991babeee025b933ee86fc873fb8f6ffa0cf5ef933b07c2adb3292eb29d07: Status 404 returned error can't find the container with id e93991babeee025b933ee86fc873fb8f6ffa0cf5ef933b07c2adb3292eb29d07 Oct 30 18:20:58.484506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:58.484423 4582 patch_prober.go:29] interesting pod/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:20:58.484506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:58.484476 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 18:20:58.731897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:58.731791 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" event={"ID":"063e6cb4-e1df-435b-a1ec-fad68c9341db","Type":"ContainerStarted","Data":"baac27e5e42fcf700b7f89f34ebed62fe55f884ee850d68f7d2588a9a7f7de18"} Oct 30 18:20:58.731897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:58.731859 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" event={"ID":"063e6cb4-e1df-435b-a1ec-fad68c9341db","Type":"ContainerStarted","Data":"e93991babeee025b933ee86fc873fb8f6ffa0cf5ef933b07c2adb3292eb29d07"} Oct 30 18:20:58.792528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:58.792425 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" podStartSLOduration=1.792410518 podStartE2EDuration="1.792410518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:58.788954342 +0000 UTC m=+1355.613348635" watchObservedRunningTime="2026-10-30 18:20:58.792410518 +0000 UTC m=+1355.616804810" Oct 30 18:20:59.484744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:59.484710 4582 patch_prober.go:29] interesting pod/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:20:59.485178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:59.484762 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 18:20:59.740636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:59.740556 4582 generic.go:356] "Generic (PLEG): container finished" podID="063e6cb4-e1df-435b-a1ec-fad68c9341db" containerID="baac27e5e42fcf700b7f89f34ebed62fe55f884ee850d68f7d2588a9a7f7de18" exitCode=0 Oct 30 18:20:59.740779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:59.740751 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" event={"ID":"063e6cb4-e1df-435b-a1ec-fad68c9341db","Type":"ContainerDied","Data":"baac27e5e42fcf700b7f89f34ebed62fe55f884ee850d68f7d2588a9a7f7de18"} Oct 30 18:20:59.966874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:20:59.963996 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:00.484973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.484800 4582 patch_prober.go:29] interesting pod/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:21:00.484973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.484890 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 18:21:00.569241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.568965 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:00.928896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.928874 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:21:00.996187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.996150 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-xks-ca\") pod \"063e6cb4-e1df-435b-a1ec-fad68c9341db\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " Oct 30 18:21:00.996368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.996225 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-openshift-ca\") pod \"063e6cb4-e1df-435b-a1ec-fad68c9341db\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " Oct 30 18:21:00.996368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.996300 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063e6cb4-e1df-435b-a1ec-fad68c9341db-kube-api-access-n47ls\" (UniqueName: \"kubernetes.io/projected/063e6cb4-e1df-435b-a1ec-fad68c9341db-kube-api-access-n47ls\") pod \"063e6cb4-e1df-435b-a1ec-fad68c9341db\" (UID: \"063e6cb4-e1df-435b-a1ec-fad68c9341db\") " Oct 30 18:21:00.996622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.996598 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-openshift-ca" pod "063e6cb4-e1df-435b-a1ec-fad68c9341db" (UID: "063e6cb4-e1df-435b-a1ec-fad68c9341db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:00.997103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.997079 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:00.998642 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.998618 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063e6cb4-e1df-435b-a1ec-fad68c9341db-kube-api-access-n47ls" pod "063e6cb4-e1df-435b-a1ec-fad68c9341db" (UID: "063e6cb4-e1df-435b-a1ec-fad68c9341db"). InnerVolumeSpecName "kube-api-access-n47ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:00.998812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:00.998789 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-xks-ca" pod "063e6cb4-e1df-435b-a1ec-fad68c9341db" (UID: "063e6cb4-e1df-435b-a1ec-fad68c9341db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:01.098102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.098073 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063e6cb4-e1df-435b-a1ec-fad68c9341db-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.098102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.098098 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47ls\" (UniqueName: \"kubernetes.io/projected/063e6cb4-e1df-435b-a1ec-fad68c9341db-kube-api-access-n47ls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:01.488469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.488439 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:21:01.488988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.488967 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:21:01.489692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.489672 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:21:01.758989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.758961 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" Oct 30 18:21:01.759174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.758961 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh" event={"ID":"063e6cb4-e1df-435b-a1ec-fad68c9341db","Type":"ContainerDied","Data":"e93991babeee025b933ee86fc873fb8f6ffa0cf5ef933b07c2adb3292eb29d07"} Oct 30 18:21:01.759174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:01.759108 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e93991babeee025b933ee86fc873fb8f6ffa0cf5ef933b07c2adb3292eb29d07" Oct 30 18:21:06.226040 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:06.226000 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:21:07.028374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:07.028341 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw"] Oct 30 18:21:07.037773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:07.037742 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db99a5d3-hvvlw"] Oct 30 18:21:07.301765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:07.301679 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e621b0f-72ac-44a2-abe7-a7a72850f6c4" path="/var/lib/kubelet/pods/5e621b0f-72ac-44a2-abe7-a7a72850f6c4/volumes" Oct 30 18:21:13.960345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:13.960312 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:14.094099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:14.094063 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:14.687877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:14.687842 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:18.029055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:18.029010 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl"] Oct 30 18:21:18.038266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:18.038241 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0a51d8-jntpl"] Oct 30 18:21:19.311220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:19.311142 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ffac5d-a35b-496d-98b3-c7e41dbf023f" path="/var/lib/kubelet/pods/11ffac5d-a35b-496d-98b3-c7e41dbf023f/volumes" Oct 30 18:21:20.341037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:20.341005 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:29.334573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.334540 4582 scope.go:122] "RemoveContainer" containerID="337499cf1c8bf76f97906af6cf791527c83aea8653c75e0410dd65000f867de3" Oct 30 18:21:29.344906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.344881 4582 scope.go:122] "RemoveContainer" containerID="7b5e2bc5c52096d10bf34c36e5ca1310a8ec7fd693c19c51d08bb7e49634d5fa" Oct 30 18:21:29.356713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.356691 4582 scope.go:122] "RemoveContainer" containerID="05d822a86f0403bc7268c7d9178a2ec5866dae1036e0f215a1f3e4fef5a848df" Oct 30 18:21:29.366170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.366147 4582 scope.go:122] "RemoveContainer" containerID="d1a308a13bc9d618a06095a9147e2e941f96e939add93d29d73c10c94d9986d7" Oct 30 18:21:29.375016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.374998 4582 scope.go:122] "RemoveContainer" containerID="93e4ed4e671013fece0c50c433d2a40aa3044a304eeb220693da5f830f689959" Oct 30 18:21:29.383584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.383561 4582 scope.go:122] "RemoveContainer" containerID="b61814cfe4ae5ade5f646ee46308779ad3dd86a64a2de1e0f2b65e9df719cf9b" Oct 30 18:21:29.391928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.391910 4582 scope.go:122] "RemoveContainer" containerID="bf8bd2318455a89e6e4727d0cf9bc422a945626f59b6fb827a56aacf8f4f0709" Oct 30 18:21:29.400343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.400323 4582 scope.go:122] "RemoveContainer" containerID="9c13cfc9e9dbea6ae70fe162a4af5b494fbd5b69338b60dfa38baf6b165f5354" Oct 30 18:21:29.408703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:29.408687 4582 scope.go:122] "RemoveContainer" containerID="2c5223476734d5940d319b0f4675957aaa8d9542471ea37dae42a5f47dfee356" Oct 30 18:21:32.684262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:32.684227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:34.590947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:34.590919 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:34.757965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:34.757932 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:34.857941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:34.857810 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:35.083403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:35.083351 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44"] Oct 30 18:21:35.084078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:35.084045 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" containerID="cri-o://33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5" gracePeriod=30 Oct 30 18:21:35.119499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:35.119439 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:35.214544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:35.214481 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:35.338845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:35.332762 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:36.227753 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:36.227719 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:21:37.075340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:37.075294 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:38.450431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.450402 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7"] Oct 30 18:21:38.450924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.450584 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" containerID="cri-o://194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d" gracePeriod=30 Oct 30 18:21:38.459845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.459797 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft"] Oct 30 18:21:38.460036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.460015 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" containerID="cri-o://504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9" gracePeriod=30 Oct 30 18:21:38.463937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.463913 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg"] Oct 30 18:21:38.464127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.464107 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" containerID="cri-o://826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994" gracePeriod=30 Oct 30 18:21:38.469614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.469586 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9"] Oct 30 18:21:38.469776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.469753 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" containerID="cri-o://8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e" gracePeriod=30 Oct 30 18:21:38.871964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:38.871928 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:40.383762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.383735 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:21:40.553720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.553686 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-podinfo\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.553933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.553762 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-kube-api-access-crl5n\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-kube-api-access-crl5n\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.553933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.553806 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-envoy\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.553933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.553899 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-socket\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.554102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.553976 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-token\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.554185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554157 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-certs\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.554238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554193 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57842deb-1907-49f8-a94c-e8a097cf4dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57842deb-1907-49f8-a94c-e8a097cf4dbf-istiod-ca-cert\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.554309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554294 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-data\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.554402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554390 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-credential-socket\") pod \"57842deb-1907-49f8-a94c-e8a097cf4dbf\" (UID: \"57842deb-1907-49f8-a94c-e8a097cf4dbf\") " Oct 30 18:21:40.554754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554656 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-certs" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.554754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554741 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57842deb-1907-49f8-a94c-e8a097cf4dbf-istiod-ca-cert" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:40.554944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554916 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-socket" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.555073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.554983 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-data" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.555073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.555053 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-credential-socket" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.555792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.555604 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.555792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.555632 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.555792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.555743 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.555792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.555759 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57842deb-1907-49f8-a94c-e8a097cf4dbf-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.555792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.555772 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.558526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.558484 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-envoy" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:40.558689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.558556 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-kube-api-access-crl5n" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "kube-api-access-crl5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:40.558855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.558832 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-podinfo" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:40.559249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.559213 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-token" pod "57842deb-1907-49f8-a94c-e8a097cf4dbf" (UID: "57842deb-1907-49f8-a94c-e8a097cf4dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:40.657541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.657505 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.657541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.657532 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.657541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.657541 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57842deb-1907-49f8-a94c-e8a097cf4dbf-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:40.657752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:40.657550 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crl5n\" (UniqueName: \"kubernetes.io/projected/57842deb-1907-49f8-a94c-e8a097cf4dbf-kube-api-access-crl5n\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:41.084693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.084612 4582 generic.go:356] "Generic (PLEG): container finished" podID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerID="33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5" exitCode=0 Oct 30 18:21:41.084693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.084672 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" Oct 30 18:21:41.084882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.084694 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" event={"ID":"57842deb-1907-49f8-a94c-e8a097cf4dbf","Type":"ContainerDied","Data":"33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5"} Oct 30 18:21:41.084882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.084749 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44" event={"ID":"57842deb-1907-49f8-a94c-e8a097cf4dbf","Type":"ContainerDied","Data":"6ecad3ad951ea4c2669c07be4992bd7bc0f8b6373efffff7e38d1131be2c33d8"} Oct 30 18:21:41.084882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.084771 4582 scope.go:122] "RemoveContainer" containerID="33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5" Oct 30 18:21:41.095705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.095689 4582 scope.go:122] "RemoveContainer" containerID="49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578" Oct 30 18:21:41.104777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.104761 4582 scope.go:122] "RemoveContainer" containerID="33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5" Oct 30 18:21:41.105066 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:41.105048 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5\": container with ID starting with 33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5 not found: ID does not exist" containerID="33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5" Oct 30 18:21:41.105123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.105075 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5"} err="failed to get container status \"33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5\": rpc error: code = NotFound desc = could not find container \"33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5\": container with ID starting with 33584bd34c7a493cc445153d6827beb00a42b7c6c59db4a557460f38c7f013d5 not found: ID does not exist" Oct 30 18:21:41.105123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.105104 4582 scope.go:122] "RemoveContainer" containerID="49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578" Oct 30 18:21:41.105330 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:41.105314 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578\": container with ID starting with 49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578 not found: ID does not exist" containerID="49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578" Oct 30 18:21:41.105370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.105337 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578"} err="failed to get container status \"49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578\": rpc error: code = NotFound desc = could not find container \"49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578\": container with ID starting with 49bfbbcaab4d724f5d6b84e9ae421adff2ba62d243ba3b2a8b78438769896578 not found: ID does not exist" Oct 30 18:21:41.114611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.114588 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:41.139528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.139506 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44"] Oct 30 18:21:41.148957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.148938 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0c6613ea-openshift-default-6684df784f-wgq44"] Oct 30 18:21:41.302058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:41.302025 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" path="/var/lib/kubelet/pods/57842deb-1907-49f8-a94c-e8a097cf4dbf/volumes" Oct 30 18:21:42.274135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:42.274102 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:21:42.274524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:42.274164 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:21:42.939023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:42.938990 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:43.046385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.046340 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:43.219534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.219457 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl"] Oct 30 18:21:43.219704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.219681 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" containerID="cri-o://2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6" gracePeriod=30 Oct 30 18:21:43.228744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.228715 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t"] Oct 30 18:21:43.228965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.228943 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" containerID="cri-o://06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd" gracePeriod=30 Oct 30 18:21:43.856592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.856465 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:43.866540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.866497 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946302 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-certs\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946418 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-data\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946504 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-podinfo\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946598 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-credential-socket\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946661 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-kube-api-access-fgtqg\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-kube-api-access-fgtqg\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946705 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istiod-ca-cert\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946763 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-socket\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946795 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-envoy\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.946885 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-token\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-token\") pod \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\" (UID: \"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583\") " Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.947409 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-credential-socket" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.947577 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-certs" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.947699 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-data" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.949706 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.949738 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.949752 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.951441 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-socket" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.951687 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istiod-ca-cert" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.955256 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-envoy" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.955582 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-token" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.955645 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-podinfo" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:43.967075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.957594 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-kube-api-access-fgtqg" pod "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" (UID: "dfb6f406-5b39-4e3a-ac81-1ac35f4dc583"). InnerVolumeSpecName "kube-api-access-fgtqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:43.969151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.968602 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:21:43.976099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:43.975949 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:44.050262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050238 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-socket\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050286 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-kube-api-access-rxzdr\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-kube-api-access-rxzdr\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050329 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-certs\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050364 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-token\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-token\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050438 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istiod-ca-cert\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050500 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-data\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050525 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-socket" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.050717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050629 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-podinfo\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050769 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-envoy\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.050896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050878 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-credential-socket\") pod \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\" (UID: \"a39f6f6d-a76b-42c7-8576-8d233cea91f9\") " Oct 30 18:21:44.051320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.050770 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-certs" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.051320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051024 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-data" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.051320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051226 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-credential-socket" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.051320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051280 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istiod-ca-cert" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:44.051802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051782 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.051905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051803 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.051905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051813 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.051905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051860 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.051905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051871 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtqg\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-kube-api-access-fgtqg\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.051905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051881 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.051905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051893 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.051905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051901 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.052198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051915 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.052198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051929 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.052198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.051943 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.054332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.054294 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-podinfo" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:44.054443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.054389 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-envoy" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.054765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.054743 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-token" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:44.055222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.055199 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:21:44.055325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.055198 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-kube-api-access-rxzdr" pod "a39f6f6d-a76b-42c7-8576-8d233cea91f9" (UID: "a39f6f6d-a76b-42c7-8576-8d233cea91f9"). InnerVolumeSpecName "kube-api-access-rxzdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:44.058207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.058191 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:21:44.119284 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.119249 4582 generic.go:356] "Generic (PLEG): container finished" podID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerID="8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e" exitCode=0 Oct 30 18:21:44.119496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.119334 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" Oct 30 18:21:44.119496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.119341 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" event={"ID":"98b1e0ec-6439-40fe-b77b-474d327d4cc9","Type":"ContainerDied","Data":"8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e"} Oct 30 18:21:44.119496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.119446 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9" event={"ID":"98b1e0ec-6439-40fe-b77b-474d327d4cc9","Type":"ContainerDied","Data":"93506dd48ece1ca4e853ecd9cecc030e2726e9c7637518bf29631a2c857d490d"} Oct 30 18:21:44.119496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.119483 4582 scope.go:122] "RemoveContainer" containerID="8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e" Oct 30 18:21:44.120894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.120872 4582 generic.go:356] "Generic (PLEG): container finished" podID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerID="504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9" exitCode=0 Oct 30 18:21:44.121024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.120917 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" Oct 30 18:21:44.121093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.121068 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" event={"ID":"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583","Type":"ContainerDied","Data":"504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9"} Oct 30 18:21:44.121150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.121102 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft" event={"ID":"dfb6f406-5b39-4e3a-ac81-1ac35f4dc583","Type":"ContainerDied","Data":"9b3d1f691233dfc9169884c2b219ab6e269351791e5a500bdc18537e233d7da0"} Oct 30 18:21:44.129848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.122960 4582 generic.go:356] "Generic (PLEG): container finished" podID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerID="826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994" exitCode=0 Oct 30 18:21:44.129848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.123079 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" event={"ID":"a39f6f6d-a76b-42c7-8576-8d233cea91f9","Type":"ContainerDied","Data":"826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994"} Oct 30 18:21:44.129848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.123119 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" event={"ID":"a39f6f6d-a76b-42c7-8576-8d233cea91f9","Type":"ContainerDied","Data":"597efca10c495217629e667e7d1ba536ff402bedd3388a1db573ec7114bd04b1"} Oct 30 18:21:44.129848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.123194 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg" Oct 30 18:21:44.130573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.130548 4582 generic.go:356] "Generic (PLEG): container finished" podID="ebb73e19-2022-4912-a893-5f32fca609d9" containerID="194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d" exitCode=0 Oct 30 18:21:44.130691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.130596 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" event={"ID":"ebb73e19-2022-4912-a893-5f32fca609d9","Type":"ContainerDied","Data":"194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d"} Oct 30 18:21:44.130691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.130633 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" event={"ID":"ebb73e19-2022-4912-a893-5f32fca609d9","Type":"ContainerDied","Data":"4c1cd4dab37207f99761b7dc58245af23cae049bc8872ee757b4bc4dc0d1a35b"} Oct 30 18:21:44.130691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.130685 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7" Oct 30 18:21:44.132500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.132278 4582 scope.go:122] "RemoveContainer" containerID="434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f" Oct 30 18:21:44.148268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.148254 4582 scope.go:122] "RemoveContainer" containerID="8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e" Oct 30 18:21:44.148592 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:44.148566 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e\": container with ID starting with 8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e not found: ID does not exist" containerID="8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e" Oct 30 18:21:44.148691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.148597 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e"} err="failed to get container status \"8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e\": rpc error: code = NotFound desc = could not find container \"8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e\": container with ID starting with 8ac6e8f656ee551da5f0ae4bd44e9b18ba670fde2af6a090a053546fd43c1f7e not found: ID does not exist" Oct 30 18:21:44.148691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.148614 4582 scope.go:122] "RemoveContainer" containerID="434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f" Oct 30 18:21:44.148911 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:44.148889 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f\": container with ID starting with 434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f not found: ID does not exist" containerID="434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f" Oct 30 18:21:44.148992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.148917 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f"} err="failed to get container status \"434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f\": rpc error: code = NotFound desc = could not find container \"434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f\": container with ID starting with 434ff3af67838db14099b6f2efb72001760166229e6fbd3e18df640247c1cd5f not found: ID does not exist" Oct 30 18:21:44.148992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.148938 4582 scope.go:122] "RemoveContainer" containerID="504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9" Oct 30 18:21:44.152863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.152837 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-podinfo\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.153061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.152927 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-data\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.153061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.152977 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-token\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-token\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.153061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153024 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb73e19-2022-4912-a893-5f32fca609d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb73e19-2022-4912-a893-5f32fca609d9-istiod-ca-cert\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153088 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-socket\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153176 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-socket\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153189 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-data" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153207 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-credential-socket\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153254 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-certs\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153307 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istiod-ca-cert\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153344 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-certs\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153383 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-socket" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153401 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-kube-api-access-st24w\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-kube-api-access-st24w\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153415 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb73e19-2022-4912-a893-5f32fca609d9-istiod-ca-cert" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153438 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-kube-api-access-llt4h\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-kube-api-access-llt4h\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153487 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-data\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153528 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-envoy\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153568 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-certs" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153582 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebb73e19-2022-4912-a893-5f32fca609d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb73e19-2022-4912-a893-5f32fca609d9-istio-podinfo\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153687 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-envoy\") pod \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\" (UID: \"98b1e0ec-6439-40fe-b77b-474d327d4cc9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153717 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-certs" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153746 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-credential-socket\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153768 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-socket" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.153790 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-istio-token\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-istio-token\") pod \"ebb73e19-2022-4912-a893-5f32fca609d9\" (UID: \"ebb73e19-2022-4912-a893-5f32fca609d9\") " Oct 30 18:21:44.154214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.154162 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istiod-ca-cert" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:44.155344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.154235 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-credential-socket" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.155344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.154719 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-data" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.155458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155410 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155435 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb73e19-2022-4912-a893-5f32fca609d9-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155450 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzdr\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-kube-api-access-rxzdr\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155466 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155481 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155495 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155508 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155522 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155535 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155549 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155563 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155575 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.155623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155589 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f6f6d-a76b-42c7-8576-8d233cea91f9-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.156207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.155834 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-credential-socket" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.156207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.156156 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-podinfo" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:44.158248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.158223 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-kube-api-access-st24w" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "kube-api-access-st24w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:44.158335 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.158260 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-token" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:44.158398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.158384 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-istio-token" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:44.159183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.159158 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-kube-api-access-llt4h" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "kube-api-access-llt4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:44.159251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.159187 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-envoy" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.160546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.160521 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-envoy" pod "98b1e0ec-6439-40fe-b77b-474d327d4cc9" (UID: "98b1e0ec-6439-40fe-b77b-474d327d4cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:44.160652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.160635 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebb73e19-2022-4912-a893-5f32fca609d9-istio-podinfo" pod "ebb73e19-2022-4912-a893-5f32fca609d9" (UID: "ebb73e19-2022-4912-a893-5f32fca609d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:44.162301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.162281 4582 scope.go:122] "RemoveContainer" containerID="504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9" Oct 30 18:21:44.162664 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:44.162645 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9\": container with ID starting with 504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9 not found: ID does not exist" containerID="504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9" Oct 30 18:21:44.162741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.162674 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9"} err="failed to get container status \"504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9\": rpc error: code = NotFound desc = could not find container \"504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9\": container with ID starting with 504e98fb570ec20ab11faeaa0a6da06f3894fb640b7fdaabcd9fbd0f70ea3ce9 not found: ID does not exist" Oct 30 18:21:44.162741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.162697 4582 scope.go:122] "RemoveContainer" containerID="826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994" Oct 30 18:21:44.172995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.172969 4582 scope.go:122] "RemoveContainer" containerID="826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994" Oct 30 18:21:44.173276 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:44.173254 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994\": container with ID starting with 826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994 not found: ID does not exist" containerID="826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994" Oct 30 18:21:44.173364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.173285 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994"} err="failed to get container status \"826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994\": rpc error: code = NotFound desc = could not find container \"826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994\": container with ID starting with 826e454319496d8a33796e22257b117be835e3d4c540e82ba51440386487d994 not found: ID does not exist" Oct 30 18:21:44.173364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.173306 4582 scope.go:122] "RemoveContainer" containerID="194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d" Oct 30 18:21:44.183382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.183340 4582 scope.go:122] "RemoveContainer" containerID="194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d" Oct 30 18:21:44.183702 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:44.183675 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d\": container with ID starting with 194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d not found: ID does not exist" containerID="194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d" Oct 30 18:21:44.183782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.183707 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d"} err="failed to get container status \"194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d\": rpc error: code = NotFound desc = could not find container \"194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d\": container with ID starting with 194c0e4ce75337aa2ea8f364ab8358a0210d06808bce6643f0ded9feb4fdf32d not found: ID does not exist" Oct 30 18:21:44.201655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.201624 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg"] Oct 30 18:21:44.207342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.207313 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zd6pg"] Oct 30 18:21:44.243385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.243360 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft"] Oct 30 18:21:44.262040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.260928 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-zvzft"] Oct 30 18:21:44.264280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264255 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st24w\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-kube-api-access-st24w\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264278 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llt4h\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-kube-api-access-llt4h\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264292 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264302 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb73e19-2022-4912-a893-5f32fca609d9-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264310 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264318 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb73e19-2022-4912-a893-5f32fca609d9-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264325 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb73e19-2022-4912-a893-5f32fca609d9-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264334 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.264489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.264342 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98b1e0ec-6439-40fe-b77b-474d327d4cc9-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:44.463699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.463647 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9"] Oct 30 18:21:44.468236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.468212 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-9qrq9"] Oct 30 18:21:44.493989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.493965 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7"] Oct 30 18:21:44.502016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:44.501994 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-5gfg7"] Oct 30 18:21:45.085606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:45.085580 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:45.301061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:45.301030 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" path="/var/lib/kubelet/pods/98b1e0ec-6439-40fe-b77b-474d327d4cc9/volumes" Oct 30 18:21:45.301507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:45.301493 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" path="/var/lib/kubelet/pods/a39f6f6d-a76b-42c7-8576-8d233cea91f9/volumes" Oct 30 18:21:45.301956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:45.301943 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" path="/var/lib/kubelet/pods/dfb6f406-5b39-4e3a-ac81-1ac35f4dc583/volumes" Oct 30 18:21:45.302676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:45.302662 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" path="/var/lib/kubelet/pods/ebb73e19-2022-4912-a893-5f32fca609d9/volumes" Oct 30 18:21:47.572392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:47.572357 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:48.165531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.165500 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:48.278010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.277982 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:48.503913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.503859 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:21:48.548190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.548162 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:21:48.630831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.630788 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-certs\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.630875 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-token\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-token\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.630917 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istiod-ca-cert\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.630945 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istiod-ca-cert\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631016 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-envoy\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631068 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-kube-api-access-25wnk\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-kube-api-access-25wnk\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631109 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-envoy\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631123 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-certs" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631149 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-credential-socket\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631200 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-data\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631258 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-socket\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631295 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-data\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631333 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-credential-socket\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631407 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istiod-ca-cert" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631502 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-podinfo\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631546 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-certs\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631586 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-socket\") pod \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\" (UID: \"10a0df4c-9172-4457-95c8-e47ed3da8cf3\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631617 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-kube-api-access-gh8t7\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-kube-api-access-gh8t7\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631653 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-token\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-token\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631680 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-podinfo\") pod \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\" (UID: \"87ef62a5-21f7-4ef5-9980-1057fcebc3fa\") " Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631685 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-data" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.631741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631737 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istiod-ca-cert" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:48.632419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631757 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-credential-socket" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.632419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631902 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-certs" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.632419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.631966 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-credential-socket" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.632618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.632579 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-socket" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.632776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.632755 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-data" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.632871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.632802 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-socket" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.633113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.633089 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.633195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.633120 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.633195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.633135 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.633195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.633148 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.633195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.633165 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.633195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.633179 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.633195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.633193 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.635200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.635149 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-kube-api-access-gh8t7" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "kube-api-access-gh8t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:48.635620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.635589 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-podinfo" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:48.635738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.635628 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-token" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:48.635997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.635902 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-envoy" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.635997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.635992 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-podinfo" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:21:48.636292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.636119 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-token" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:48.636292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.636187 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-envoy" pod "87ef62a5-21f7-4ef5-9980-1057fcebc3fa" (UID: "87ef62a5-21f7-4ef5-9980-1057fcebc3fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:21:48.637398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.637376 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-kube-api-access-25wnk" pod "10a0df4c-9172-4457-95c8-e47ed3da8cf3" (UID: "10a0df4c-9172-4457-95c8-e47ed3da8cf3"). InnerVolumeSpecName "kube-api-access-25wnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:48.734920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734845 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.734920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734878 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.734920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734892 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wnk\" (UniqueName: \"kubernetes.io/projected/10a0df4c-9172-4457-95c8-e47ed3da8cf3-kube-api-access-25wnk\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.734920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734910 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.734920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734923 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.735194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734937 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.735194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734951 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a0df4c-9172-4457-95c8-e47ed3da8cf3-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.735194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734964 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a0df4c-9172-4457-95c8-e47ed3da8cf3-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.735194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734979 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh8t7\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-kube-api-access-gh8t7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.735194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.734992 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:48.735194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:48.735005 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef62a5-21f7-4ef5-9980-1057fcebc3fa-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:21:49.173940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.173903 4582 generic.go:356] "Generic (PLEG): container finished" podID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerID="2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6" exitCode=0 Oct 30 18:21:49.174109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.173951 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" Oct 30 18:21:49.174109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.173989 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" event={"ID":"87ef62a5-21f7-4ef5-9980-1057fcebc3fa","Type":"ContainerDied","Data":"2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6"} Oct 30 18:21:49.174109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.174041 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl" event={"ID":"87ef62a5-21f7-4ef5-9980-1057fcebc3fa","Type":"ContainerDied","Data":"bfb5998d3c4372c7b6adf2c83b57350cf6a034ca442566d4a1b746eb822a2da9"} Oct 30 18:21:49.174109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.174064 4582 scope.go:122] "RemoveContainer" containerID="2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6" Oct 30 18:21:49.175563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.175542 4582 generic.go:356] "Generic (PLEG): container finished" podID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerID="06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd" exitCode=0 Oct 30 18:21:49.175676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.175592 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" Oct 30 18:21:49.175676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.175625 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" event={"ID":"10a0df4c-9172-4457-95c8-e47ed3da8cf3","Type":"ContainerDied","Data":"06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd"} Oct 30 18:21:49.175789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.175675 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t" event={"ID":"10a0df4c-9172-4457-95c8-e47ed3da8cf3","Type":"ContainerDied","Data":"c3a2ca4f6a67ac183d013c170e319f0f52ac60f137949933ed4fd35fdf6d124f"} Oct 30 18:21:49.185498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.185438 4582 scope.go:122] "RemoveContainer" containerID="2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6" Oct 30 18:21:49.185724 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:49.185707 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6\": container with ID starting with 2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6 not found: ID does not exist" containerID="2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6" Oct 30 18:21:49.185804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.185732 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6"} err="failed to get container status \"2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6\": rpc error: code = NotFound desc = could not find container \"2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6\": container with ID starting with 2c6db287b4b59f6e9b0c6966a157dfcab060451526fab4e0f5ce1e7770cf19d6 not found: ID does not exist" Oct 30 18:21:49.185804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.185748 4582 scope.go:122] "RemoveContainer" containerID="06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd" Oct 30 18:21:49.195366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.195345 4582 scope.go:122] "RemoveContainer" containerID="06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd" Oct 30 18:21:49.195619 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:21:49.195601 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd\": container with ID starting with 06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd not found: ID does not exist" containerID="06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd" Oct 30 18:21:49.195684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.195625 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd"} err="failed to get container status \"06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd\": rpc error: code = NotFound desc = could not find container \"06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd\": container with ID starting with 06440904a613518d56d2a0db183b617f387306ccfd7055d8cc78af85a593f2cd not found: ID does not exist" Oct 30 18:21:49.220918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.220895 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl"] Oct 30 18:21:49.230119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.230097 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-wrwbl"] Oct 30 18:21:49.257683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.257654 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t"] Oct 30 18:21:49.260837 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.260802 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zzc6t"] Oct 30 18:21:49.301554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.301527 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" path="/var/lib/kubelet/pods/10a0df4c-9172-4457-95c8-e47ed3da8cf3/volumes" Oct 30 18:21:49.302050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.302035 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" path="/var/lib/kubelet/pods/87ef62a5-21f7-4ef5-9980-1057fcebc3fa/volumes" Oct 30 18:21:49.683933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.683901 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:49.876574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.876543 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:49.978833 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:49.978735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:51.278755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:51.278714 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:52.003618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:52.003577 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:52.117534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:52.117495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:52.181688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:52.181655 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh"] Oct 30 18:21:52.190323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:52.190289 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ee874-hh6jh"] Oct 30 18:21:52.264258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:52.263180 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:52.656139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:52.656102 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:53.301019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:53.300988 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063e6cb4-e1df-435b-a1ec-fad68c9341db" path="/var/lib/kubelet/pods/063e6cb4-e1df-435b-a1ec-fad68c9341db/volumes" Oct 30 18:21:54.906286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:54.906255 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:54.995451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:54.995421 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:55.091423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:55.091393 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w"] Oct 30 18:21:55.091634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:55.091605 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" containerID="cri-o://f264d1f045b41f5607d08017ce6cc7a73da01e65589855eb642b4702a68f0171" gracePeriod=30 Oct 30 18:21:55.093816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:55.093787 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:57.137924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:57.137882 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:58.509833 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:58.509784 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:58.616619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:58.616581 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:58.846201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:58.846121 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:59.563410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:21:59.563373 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:00.325802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.325769 4582 generic.go:356] "Generic (PLEG): container finished" podID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerID="f264d1f045b41f5607d08017ce6cc7a73da01e65589855eb642b4702a68f0171" exitCode=0 Oct 30 18:22:00.325966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.325855 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" event={"ID":"63bed21e-b5ec-4411-b09c-456575cd00fc","Type":"ContainerDied","Data":"f264d1f045b41f5607d08017ce6cc7a73da01e65589855eb642b4702a68f0171"} Oct 30 18:22:00.367045 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.366989 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:22:00.404295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404269 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-podinfo\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404344 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-certs\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404405 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-envoy\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404472 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-kube-api-access-rb7fg\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-kube-api-access-rb7fg\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404535 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-socket\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404602 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-data\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404614 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-certs" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:00.404748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404702 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-credential-socket\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404745 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bed21e-b5ec-4411-b09c-456575cd00fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bed21e-b5ec-4411-b09c-456575cd00fc-istiod-ca-cert\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404785 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-token\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-token\") pod \"63bed21e-b5ec-4411-b09c-456575cd00fc\" (UID: \"63bed21e-b5ec-4411-b09c-456575cd00fc\") " Oct 30 18:22:00.404945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404861 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-socket" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:00.404996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404934 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-data" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:00.404996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.404982 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-credential-socket" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:00.405446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.405173 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bed21e-b5ec-4411-b09c-456575cd00fc-istiod-ca-cert" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:00.405446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.405389 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.405446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.405403 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bed21e-b5ec-4411-b09c-456575cd00fc-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.405446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.405413 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.405446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.405421 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.405446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.405436 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.406711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.406683 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-envoy" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:00.406966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.406937 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-podinfo" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:00.407062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.406967 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-kube-api-access-rb7fg" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "kube-api-access-rb7fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:00.407062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.406989 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-token" pod "63bed21e-b5ec-4411-b09c-456575cd00fc" (UID: "63bed21e-b5ec-4411-b09c-456575cd00fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:00.506476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.506445 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.506476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.506472 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.506476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.506481 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bed21e-b5ec-4411-b09c-456575cd00fc-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.506704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:00.506492 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7fg\" (UniqueName: \"kubernetes.io/projected/63bed21e-b5ec-4411-b09c-456575cd00fc-kube-api-access-rb7fg\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:01.333778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:01.333741 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" event={"ID":"63bed21e-b5ec-4411-b09c-456575cd00fc","Type":"ContainerDied","Data":"17f4df9b04402e36940f681ece83d70c1ec5fad9918502ff6f8def38a189983b"} Oct 30 18:22:01.334195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:01.333801 4582 scope.go:122] "RemoveContainer" containerID="f264d1f045b41f5607d08017ce6cc7a73da01e65589855eb642b4702a68f0171" Oct 30 18:22:01.334195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:01.333756 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w" Oct 30 18:22:01.389390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:01.389344 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:01.389848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:01.389814 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w"] Oct 30 18:22:01.393122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:01.393102 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-7ws4w"] Oct 30 18:22:03.149569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:03.149536 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:03.231999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:03.231968 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx"] Oct 30 18:22:03.232188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:03.232164 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" containerID="cri-o://4ed0023c60e659eaa281b7ce42613927247ac4dbe91babe39d92cb13c38293e9" gracePeriod=30 Oct 30 18:22:03.239561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:03.239535 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x"] Oct 30 18:22:03.239737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:03.239711 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" containerID="cri-o://91f1852936cd5c4fb7f928265377fb71ee85449cd790fad7434c248f9ffe789e" gracePeriod=30 Oct 30 18:22:03.303936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:03.303900 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" path="/var/lib/kubelet/pods/63bed21e-b5ec-4411-b09c-456575cd00fc/volumes" Oct 30 18:22:04.949943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:04.949913 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:06.225091 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:22:06.225056 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:22:06.711394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:06.711347 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:08.397169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.397133 4582 generic.go:356] "Generic (PLEG): container finished" podID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerID="91f1852936cd5c4fb7f928265377fb71ee85449cd790fad7434c248f9ffe789e" exitCode=0 Oct 30 18:22:08.397851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.397211 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" event={"ID":"d92b1954-580c-4d33-8fd9-e9e54ada9670","Type":"ContainerDied","Data":"91f1852936cd5c4fb7f928265377fb71ee85449cd790fad7434c248f9ffe789e"} Oct 30 18:22:08.399836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.399792 4582 generic.go:356] "Generic (PLEG): container finished" podID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerID="4ed0023c60e659eaa281b7ce42613927247ac4dbe91babe39d92cb13c38293e9" exitCode=0 Oct 30 18:22:08.399942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.399891 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" event={"ID":"2e1585b9-efd1-4cc2-8643-522e787e6f69","Type":"ContainerDied","Data":"4ed0023c60e659eaa281b7ce42613927247ac4dbe91babe39d92cb13c38293e9"} Oct 30 18:22:08.518112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.518087 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:22:08.553052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.553029 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:22:08.620388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620350 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-socket\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.620588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620406 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-data\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.620588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620448 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-credential-socket\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.620588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620499 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-token\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-token\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.620588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620540 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-kube-api-access-wvrzh\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-kube-api-access-wvrzh\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.620588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620577 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-data\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.620869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620623 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-kube-api-access-8csvz\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-kube-api-access-8csvz\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.620869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620675 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92b1954-580c-4d33-8fd9-e9e54ada9670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92b1954-580c-4d33-8fd9-e9e54ada9670-istiod-ca-cert\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.620869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.620739 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-credential-socket\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.622449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.622414 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-credential-socket" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.622601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.622443 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-socket" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.622601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.622428 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-data" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.622798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.622776 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92b1954-580c-4d33-8fd9-e9e54ada9670-istiod-ca-cert" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:08.623096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.623037 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-credential-socket" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.623261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.623229 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-data" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.623778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.623752 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-certs" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.624662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.624636 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-token" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:08.624873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.624803 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-kube-api-access-wvrzh" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "kube-api-access-wvrzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:08.625557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625220 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-certs\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.625557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625345 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-podinfo\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.625557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625484 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1585b9-efd1-4cc2-8643-522e787e6f69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1585b9-efd1-4cc2-8643-522e787e6f69-istiod-ca-cert\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.625557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625521 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-envoy\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.625557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625550 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-podinfo\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.625913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625627 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-kube-api-access-8csvz" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "kube-api-access-8csvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:08.625913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625650 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-token\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-token\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.625913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625690 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-envoy\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.625913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625748 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-socket\") pod \"2e1585b9-efd1-4cc2-8643-522e787e6f69\" (UID: \"2e1585b9-efd1-4cc2-8643-522e787e6f69\") " Oct 30 18:22:08.625913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.625874 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-certs\") pod \"d92b1954-580c-4d33-8fd9-e9e54ada9670\" (UID: \"d92b1954-580c-4d33-8fd9-e9e54ada9670\") " Oct 30 18:22:08.626189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.626113 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1585b9-efd1-4cc2-8643-522e787e6f69-istiod-ca-cert" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:08.626587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.626560 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-certs" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.626794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.626769 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-socket" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.627252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627221 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627257 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627275 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1585b9-efd1-4cc2-8643-522e787e6f69-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627297 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627317 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627332 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627347 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627367 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627382 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627398 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrzh\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-kube-api-access-wvrzh\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627414 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627434 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8csvz\" (UniqueName: \"kubernetes.io/projected/d92b1954-580c-4d33-8fd9-e9e54ada9670-kube-api-access-8csvz\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.627687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.627449 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92b1954-580c-4d33-8fd9-e9e54ada9670-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.628179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.628141 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-podinfo" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:08.628361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.628335 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-envoy" pod "d92b1954-580c-4d33-8fd9-e9e54ada9670" (UID: "d92b1954-580c-4d33-8fd9-e9e54ada9670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.628890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.628854 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-envoy" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:08.629167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.629148 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-podinfo" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:08.630549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.630528 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-token" pod "2e1585b9-efd1-4cc2-8643-522e787e6f69" (UID: "2e1585b9-efd1-4cc2-8643-522e787e6f69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:08.636794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.636767 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:08.733426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.730603 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.733426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.730640 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.733426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.730657 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.733426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.730670 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1585b9-efd1-4cc2-8643-522e787e6f69-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:08.733426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:08.730684 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92b1954-580c-4d33-8fd9-e9e54ada9670-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:09.409425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.409388 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" event={"ID":"d92b1954-580c-4d33-8fd9-e9e54ada9670","Type":"ContainerDied","Data":"c49a53e677cd50997b13653cceabfc737cfa2f2e351063737175d70071c7ff0c"} Oct 30 18:22:09.409924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.409456 4582 scope.go:122] "RemoveContainer" containerID="91f1852936cd5c4fb7f928265377fb71ee85449cd790fad7434c248f9ffe789e" Oct 30 18:22:09.409924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.409397 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x" Oct 30 18:22:09.411343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.411321 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" event={"ID":"2e1585b9-efd1-4cc2-8643-522e787e6f69","Type":"ContainerDied","Data":"52e886c3088b0300e6d81194fd5eae982637570c17053daace6ba30525c34cf2"} Oct 30 18:22:09.411450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.411352 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx" Oct 30 18:22:09.421553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.421537 4582 scope.go:122] "RemoveContainer" containerID="4ed0023c60e659eaa281b7ce42613927247ac4dbe91babe39d92cb13c38293e9" Oct 30 18:22:09.447789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.447761 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl"] Oct 30 18:22:09.448345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448330 4582 cpu_manager.go:414] "containerMap: removing container" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" Oct 30 18:22:09.448394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448346 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" Oct 30 18:22:09.448394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448358 4582 cpu_manager.go:414] "containerMap: removing container" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" Oct 30 18:22:09.448394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448364 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" Oct 30 18:22:09.448394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448371 4582 cpu_manager.go:414] "containerMap: removing container" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" Oct 30 18:22:09.448394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448377 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" Oct 30 18:22:09.448394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448385 4582 cpu_manager.go:414] "containerMap: removing container" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" Oct 30 18:22:09.448394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448390 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448398 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448403 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448415 4582 cpu_manager.go:414] "containerMap: removing container" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448419 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448426 4582 cpu_manager.go:414] "containerMap: removing container" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448431 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448444 4582 cpu_manager.go:414] "containerMap: removing container" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448451 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448468 4582 cpu_manager.go:414] "containerMap: removing container" podUID="063e6cb4-e1df-435b-a1ec-fad68c9341db" containerName="revoke-keys" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448475 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063e6cb4-e1df-435b-a1ec-fad68c9341db" containerName="revoke-keys" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448489 4582 cpu_manager.go:414] "containerMap: removing container" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448495 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448512 4582 cpu_manager.go:414] "containerMap: removing container" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448520 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448554 4582 cpu_manager.go:414] "containerMap: removing container" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" Oct 30 18:22:09.448608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448559 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448658 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39f6f6d-a76b-42c7-8576-8d233cea91f9" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448668 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="063e6cb4-e1df-435b-a1ec-fad68c9341db" containerName="revoke-keys" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448674 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a0df4c-9172-4457-95c8-e47ed3da8cf3" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448680 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448687 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bed21e-b5ec-4411-b09c-456575cd00fc" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448698 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448707 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb6f406-5b39-4e3a-ac81-1ac35f4dc583" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448715 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448723 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b1e0ec-6439-40fe-b77b-474d327d4cc9" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448732 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb73e19-2022-4912-a893-5f32fca609d9" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448741 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" containerName="istio-proxy" Oct 30 18:22:09.449092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.448748 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ef62a5-21f7-4ef5-9980-1057fcebc3fa" containerName="istio-proxy" Oct 30 18:22:09.454035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.454019 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.469440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.469417 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:22:09.471398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.471370 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx"] Oct 30 18:22:09.474635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.474616 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl"] Oct 30 18:22:09.476492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.476471 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:22:09.476586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.476552 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:22:09.476645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.476599 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:22:09.479414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.479393 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-sgtlx"] Oct 30 18:22:09.509669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.509586 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x"] Oct 30 18:22:09.515692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.515662 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-85r6x"] Oct 30 18:22:09.541412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.541385 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zkd\" (UniqueName: \"kubernetes.io/projected/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-kube-api-access-x2zkd\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.541523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.541436 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.541654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.541637 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.642734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.642706 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.642893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.642791 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zkd\" (UniqueName: \"kubernetes.io/projected/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-kube-api-access-x2zkd\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.642893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.642859 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.643494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.643472 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.645158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.645134 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.658690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.658663 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zkd\" (UniqueName: \"kubernetes.io/projected/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-kube-api-access-x2zkd\") pod \"maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:09.767643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:09.767572 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:10.036751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.036670 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:10.065304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.065276 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl"] Oct 30 18:22:10.069364 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:22:10.069334 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c1ac4d_9ffa_46a5_a7e9_3a89e15a376c.slice/crio-fde1a45611e9e06ebeaf6033ca79e47d0a4b854d2fe1fbe0af76fb8b6cdb8bbb WatchSource:0}: Error finding container fde1a45611e9e06ebeaf6033ca79e47d0a4b854d2fe1fbe0af76fb8b6cdb8bbb: Status 404 returned error can't find the container with id fde1a45611e9e06ebeaf6033ca79e47d0a4b854d2fe1fbe0af76fb8b6cdb8bbb Oct 30 18:22:10.262123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.262090 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:10.423280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.423237 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" event={"ID":"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c","Type":"ContainerStarted","Data":"fde1a45611e9e06ebeaf6033ca79e47d0a4b854d2fe1fbe0af76fb8b6cdb8bbb"} Oct 30 18:22:10.800476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.800442 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4"] Oct 30 18:22:10.801418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.801393 4582 cpu_manager.go:414] "containerMap: removing container" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" Oct 30 18:22:10.801418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.801417 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" Oct 30 18:22:10.801613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.801600 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="57842deb-1907-49f8-a94c-e8a097cf4dbf" containerName="istio-proxy" Oct 30 18:22:10.805657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.805639 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:10.830681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.827023 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4"] Oct 30 18:22:10.831583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.830855 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-011d6f\"" Oct 30 18:22:10.831583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.830867 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-011d6f-dockercfg-lpbvs\"" Oct 30 18:22:10.963423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.963382 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b9f17f-a075-4f44-88c1-30658d0eaaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:10.963634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:10.963610 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmmf\" (UniqueName: \"kubernetes.io/projected/18b9f17f-a075-4f44-88c1-30658d0eaaed-kube-api-access-7fmmf\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:11.065015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.064917 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b9f17f-a075-4f44-88c1-30658d0eaaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:11.065182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.065082 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmmf\" (UniqueName: \"kubernetes.io/projected/18b9f17f-a075-4f44-88c1-30658d0eaaed-kube-api-access-7fmmf\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:11.065688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.065669 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b9f17f-a075-4f44-88c1-30658d0eaaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:11.081220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.081191 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmmf\" (UniqueName: \"kubernetes.io/projected/18b9f17f-a075-4f44-88c1-30658d0eaaed-kube-api-access-7fmmf\") pod \"payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:11.120564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.120542 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:11.259413 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.259381 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt"] Oct 30 18:22:11.264486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.264457 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.280924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.280902 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt"] Oct 30 18:22:11.307090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.307057 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1585b9-efd1-4cc2-8643-522e787e6f69" path="/var/lib/kubelet/pods/2e1585b9-efd1-4cc2-8643-522e787e6f69/volumes" Oct 30 18:22:11.308181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.308157 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92b1954-580c-4d33-8fd9-e9e54ada9670" path="/var/lib/kubelet/pods/d92b1954-580c-4d33-8fd9-e9e54ada9670/volumes" Oct 30 18:22:11.369107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.369074 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd25b6ee-bc40-4927-b368-7c7433acb2db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.369250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.369193 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp69c\" (UniqueName: \"kubernetes.io/projected/dd25b6ee-bc40-4927-b368-7c7433acb2db-kube-api-access-bp69c\") pod \"payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.416924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.416897 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4"] Oct 30 18:22:11.419765 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:22:11.419729 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b9f17f_a075_4f44_88c1_30658d0eaaed.slice/crio-ccff782af1bd0fbb6f464f2633bc41db555085db5ac65390f264d2cd28efe503 WatchSource:0}: Error finding container ccff782af1bd0fbb6f464f2633bc41db555085db5ac65390f264d2cd28efe503: Status 404 returned error can't find the container with id ccff782af1bd0fbb6f464f2633bc41db555085db5ac65390f264d2cd28efe503 Oct 30 18:22:11.435235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.435201 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" event={"ID":"18b9f17f-a075-4f44-88c1-30658d0eaaed","Type":"ContainerStarted","Data":"ccff782af1bd0fbb6f464f2633bc41db555085db5ac65390f264d2cd28efe503"} Oct 30 18:22:11.437118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.437087 4582 generic.go:356] "Generic (PLEG): container finished" podID="c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" containerID="a43bfb1f9b2453f15013d2b4056300f8edffcfa8b9c537b03b8b4365eeb96735" exitCode=0 Oct 30 18:22:11.437224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.437165 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" event={"ID":"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c","Type":"ContainerDied","Data":"a43bfb1f9b2453f15013d2b4056300f8edffcfa8b9c537b03b8b4365eeb96735"} Oct 30 18:22:11.470489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.470461 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp69c\" (UniqueName: \"kubernetes.io/projected/dd25b6ee-bc40-4927-b368-7c7433acb2db-kube-api-access-bp69c\") pod \"payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.470708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.470688 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd25b6ee-bc40-4927-b368-7c7433acb2db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.471717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.471587 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd25b6ee-bc40-4927-b368-7c7433acb2db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.486924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.486847 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp69c\" (UniqueName: \"kubernetes.io/projected/dd25b6ee-bc40-4927-b368-7c7433acb2db-kube-api-access-bp69c\") pod \"payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.579233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.579202 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:11.742873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.742845 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:11.888618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:11.888590 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt"] Oct 30 18:22:12.446430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.446385 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" event={"ID":"18b9f17f-a075-4f44-88c1-30658d0eaaed","Type":"ContainerStarted","Data":"e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f"} Oct 30 18:22:12.447162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.446486 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:12.448293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.448269 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" event={"ID":"dd25b6ee-bc40-4927-b368-7c7433acb2db","Type":"ContainerStarted","Data":"8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193"} Oct 30 18:22:12.448428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.448314 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" event={"ID":"dd25b6ee-bc40-4927-b368-7c7433acb2db","Type":"ContainerStarted","Data":"5ab06ddfaefbedb7a0998154fb3d1149d2e3e6c4b15828663f9a6e987f520849"} Oct 30 18:22:12.484015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.483935 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" podStartSLOduration=2.483915628 podStartE2EDuration="2.483915628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:12.48091199 +0000 UTC m=+1429.305306285" watchObservedRunningTime="2026-10-30 18:22:12.483915628 +0000 UTC m=+1429.308309920" Oct 30 18:22:12.518027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.517907 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" podStartSLOduration=1.517892377 podStartE2EDuration="1.517892377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:12.514783415 +0000 UTC m=+1429.339177709" watchObservedRunningTime="2026-10-30 18:22:12.517892377 +0000 UTC m=+1429.342286669" Oct 30 18:22:12.624716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.624693 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:12.788539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.788458 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-openshift-ca\") pod \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " Oct 30 18:22:12.788801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.788772 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-xks-ca\") pod \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " Oct 30 18:22:12.789011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.788809 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-openshift-ca" pod "c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" (UID: "c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:12.789011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.788878 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-kube-api-access-x2zkd\" (UniqueName: \"kubernetes.io/projected/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-kube-api-access-x2zkd\") pod \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\" (UID: \"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c\") " Oct 30 18:22:12.789750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.789467 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:12.791084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.791044 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-xks-ca" pod "c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" (UID: "c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:12.792966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.792941 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-kube-api-access-x2zkd" pod "c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" (UID: "c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c"). InnerVolumeSpecName "kube-api-access-x2zkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:12.890908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.890879 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:12.890908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:12.890908 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2zkd\" (UniqueName: \"kubernetes.io/projected/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c-kube-api-access-x2zkd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:22:13.457899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:13.457866 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" Oct 30 18:22:13.458379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:13.457871 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl" event={"ID":"c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c","Type":"ContainerDied","Data":"fde1a45611e9e06ebeaf6033ca79e47d0a4b854d2fe1fbe0af76fb8b6cdb8bbb"} Oct 30 18:22:13.458379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:13.457992 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde1a45611e9e06ebeaf6033ca79e47d0a4b854d2fe1fbe0af76fb8b6cdb8bbb" Oct 30 18:22:13.458379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:13.458330 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:13.712133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:13.712099 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:14.091940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:14.091852 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:14.192681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:14.192644 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:15.505560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:15.505522 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:15.590518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:15.590486 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:15.718456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:15.718418 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:17.702217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:17.702179 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:19.617756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:19.617717 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:20.072979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:20.071776 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:20.169709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:20.169668 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:20.461790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:20.461755 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:20.841650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:20.841571 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:20.943473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:20.943443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:22.306010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:22.305979 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:23.460657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:23.460626 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:22:24.319384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:24.319352 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:24.468927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:24.468901 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:22:26.668578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:26.668542 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:28.316419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:28.316382 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:28.726657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:28.726591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:29.441267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.441219 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:29.477200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.477171 4582 scope.go:122] "RemoveContainer" containerID="4f6789448755db741c9ebc3862338a945b866dbf222f0d10488c85cf7b6b66bb" Oct 30 18:22:29.487453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.487429 4582 scope.go:122] "RemoveContainer" containerID="4e5f6af1a421f8c6caa507dbad9d348b4c479a7becb99f128c57807194ba3acf" Oct 30 18:22:29.496264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.496247 4582 scope.go:122] "RemoveContainer" containerID="7f2404d88c4a16e94de9401a14b4de9572f2cd157470be865c4092bb3ed18459" Oct 30 18:22:29.505019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.505001 4582 scope.go:122] "RemoveContainer" containerID="69541fa7e178a5fe58ae7b28d75aded0e5603fd4c3fbcf974a87c100f3cc7cd0" Oct 30 18:22:29.516424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.516403 4582 scope.go:122] "RemoveContainer" containerID="2f112be764e8f78a78f4c0eb678e87fcdda5f163a064e3906a53a09520d61fa6" Oct 30 18:22:29.525010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.524989 4582 scope.go:122] "RemoveContainer" containerID="e73d1592209c7cda95b13c9ee31e1d76ee1de47d9548b7108c430da227317a3d" Oct 30 18:22:29.534768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.534749 4582 scope.go:122] "RemoveContainer" containerID="7a430d237920865868fe66859b3fe1278ba7fbcc7a62854125d67e92ae686bd4" Oct 30 18:22:29.543145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.543126 4582 scope.go:122] "RemoveContainer" containerID="e943a67c5b696f76e6c3a6fb73bdbffba4c3266166a8ab9b3c93c3cc1e90465b" Oct 30 18:22:29.551325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.551306 4582 scope.go:122] "RemoveContainer" containerID="fc514ec76099307de77f674ab9676aa9a8b45cb6548b521675d068c86ddb5068" Oct 30 18:22:29.865065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.864986 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl"] Oct 30 18:22:29.866907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.865986 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" containerName="revoke-keys" Oct 30 18:22:29.866907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.866011 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" containerName="revoke-keys" Oct 30 18:22:29.866907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.866182 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" containerName="revoke-keys" Oct 30 18:22:29.872431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.872406 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.891045 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.891021 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0ac6bb-openshift-default-dockercfg-b62k4\"" Oct 30 18:22:29.904597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.904566 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl"] Oct 30 18:22:29.912394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.912370 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:29.914803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.914777 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istiod-ca-cert\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.914935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.914848 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-podinfo\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.914935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.914911 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-data\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.915067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.915036 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-certs\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.915195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.915172 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-token\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.915262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.915213 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-socket\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.915320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.915305 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-credential-socket\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.915417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.915401 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-envoy\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:29.915487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:29.915469 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wsns\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-kube-api-access-2wsns\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.016543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016509 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-token\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.016543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016546 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-socket\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.016854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016587 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-credential-socket\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.016854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016664 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-envoy\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.016854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016709 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wsns\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-kube-api-access-2wsns\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.016854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016774 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istiod-ca-cert\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.016854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016810 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-podinfo\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.017143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016896 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-data\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.017143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.016962 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-certs\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.017143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.017069 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-credential-socket\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.017778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.017752 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-socket\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.018406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.018361 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-data\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.018802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.018777 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-certs\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.019685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.019197 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istiod-ca-cert\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.019685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.019351 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-envoy\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.020426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.020402 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-podinfo\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.035280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.035251 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8"] Oct 30 18:22:30.035897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.035802 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-token\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.039641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.039616 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wsns\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-kube-api-access-2wsns\") pod \"e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.047213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.047187 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e3303e-2krj8"] Oct 30 18:22:30.111337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.111305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:30.199065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.199020 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:30.210593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.210521 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:30.560343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.560269 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl"] Oct 30 18:22:30.563992 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:22:30.563966 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cc9b03_2f67_445f_9e2d_be1a776ee428.slice/crio-8f00284e75e133e9c9256b988b01ef15abe9e41ad0f665cc41647b20edd2f067 WatchSource:0}: Error finding container 8f00284e75e133e9c9256b988b01ef15abe9e41ad0f665cc41647b20edd2f067: Status 404 returned error can't find the container with id 8f00284e75e133e9c9256b988b01ef15abe9e41ad0f665cc41647b20edd2f067 Oct 30 18:22:30.566712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.566677 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:22:30.566779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.566746 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:22:30.566845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.566778 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:22:30.599229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:30.599077 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" event={"ID":"a0cc9b03-2f67-445f-9e2d-be1a776ee428","Type":"ContainerStarted","Data":"8f00284e75e133e9c9256b988b01ef15abe9e41ad0f665cc41647b20edd2f067"} Oct 30 18:22:31.301896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:31.301866 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8741f0df-f42e-4dd6-952e-a67ca1c68d66" path="/var/lib/kubelet/pods/8741f0df-f42e-4dd6-952e-a67ca1c68d66/volumes" Oct 30 18:22:31.612184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:31.612099 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" event={"ID":"a0cc9b03-2f67-445f-9e2d-be1a776ee428","Type":"ContainerStarted","Data":"83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e"} Oct 30 18:22:31.657620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:31.657567 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podStartSLOduration=2.6575546279999998 podStartE2EDuration="2.657554628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:31.657138037 +0000 UTC m=+1448.481532329" watchObservedRunningTime="2026-10-30 18:22:31.657554628 +0000 UTC m=+1448.481948921" Oct 30 18:22:31.759091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:31.759060 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:31.861974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:31.861941 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:31.964691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:31.964654 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:32.210292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:32.210262 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:32.230889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:32.228391 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:32.618551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:32.618526 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:32.619512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:32.619494 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:22:34.040048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:34.040015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:34.140531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:34.140503 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.771034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:35.770999 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.873030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:35.872989 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:36.229500 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:22:36.229462 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:22:36.866737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:36.866695 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:37.817506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:37.817474 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:38.885509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:38.885476 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:40.171523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:40.171489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:40.360640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:40.360557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:41.674941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:41.674909 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:43.620805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:43.620771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:45.518100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:45.518067 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:47.405399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:47.405366 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:47.506741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:47.506704 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:47.608135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:47.608102 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:48.007607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.007572 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:48.221779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.221745 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn"] Oct 30 18:22:48.228306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.228285 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.249182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.249148 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-17df15-openshift-default-dockercfg-gwrwt\"" Oct 30 18:22:48.264311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.264252 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn"] Oct 30 18:22:48.276166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.276143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:48.309511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309484 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-certs\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309523 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-credential-socket\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309555 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a41fd-276b-4bf5-879d-300ef0977e38-istiod-ca-cert\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309670 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-data\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309715 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llvw\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-kube-api-access-7llvw\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309792 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-envoy\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309861 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-podinfo\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309898 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-token\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.309949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.309938 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-socket\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.378815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.378778 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:48.412104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412062 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-certs\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412120 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-credential-socket\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412160 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a41fd-276b-4bf5-879d-300ef0977e38-istiod-ca-cert\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412202 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-data\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412256 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llvw\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-kube-api-access-7llvw\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412323 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-envoy\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412381 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-podinfo\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412422 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-token\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.412615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.412476 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-socket\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.413363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.413314 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-socket\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.415857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.415814 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-certs\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.418427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.416703 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-credential-socket\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.418577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.417388 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a41fd-276b-4bf5-879d-300ef0977e38-istiod-ca-cert\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.418692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.417624 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-data\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.462937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.456709 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-podinfo\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.468936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.468704 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-token\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.469292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.469225 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-envoy\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.473071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.472989 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llvw\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-kube-api-access-7llvw\") pod \"e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.557154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.554504 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:48.876586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.876553 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn"] Oct 30 18:22:48.878960 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:22:48.878895 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10a41fd_276b_4bf5_879d_300ef0977e38.slice/crio-581fbd0bad4cc559ae2b8dbbf538cd21c42210a02be826fe4919cb0956e00173 WatchSource:0}: Error finding container 581fbd0bad4cc559ae2b8dbbf538cd21c42210a02be826fe4919cb0956e00173: Status 404 returned error can't find the container with id 581fbd0bad4cc559ae2b8dbbf538cd21c42210a02be826fe4919cb0956e00173 Oct 30 18:22:48.881615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.881578 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:22:48.881722 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.881691 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:22:48.881762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:48.881733 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:22:49.744473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:49.744436 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" event={"ID":"f10a41fd-276b-4bf5-879d-300ef0977e38","Type":"ContainerStarted","Data":"9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08"} Oct 30 18:22:49.744939 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:49.744496 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" event={"ID":"f10a41fd-276b-4bf5-879d-300ef0977e38","Type":"ContainerStarted","Data":"581fbd0bad4cc559ae2b8dbbf538cd21c42210a02be826fe4919cb0956e00173"} Oct 30 18:22:49.792035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:49.791986 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podStartSLOduration=1.791972458 podStartE2EDuration="1.791972458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:49.789907431 +0000 UTC m=+1466.614301723" watchObservedRunningTime="2026-10-30 18:22:49.791972458 +0000 UTC m=+1466.616366750" Oct 30 18:22:49.859998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:49.859966 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:50.555417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:50.555384 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:50.561273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:50.561247 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:50.751380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:50.751340 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:50.752356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:50.752334 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:22:52.178134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:52.178099 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:52.279394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:52.279363 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:52.396843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:52.394136 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:53.893019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:53.892984 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:54.071533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:54.071496 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:54.342389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:54.342307 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:56.197660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:56.197620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:58.189870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:58.189829 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:59.918194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:22:59.918163 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:00.002572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:00.002536 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:00.108668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:00.108632 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:01.466802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:01.466757 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:02.135140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:02.135088 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:02.236460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:02.236422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:02.291524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:02.291493 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl"] Oct 30 18:23:02.298349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:02.298321 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-de14f2-dkmrl"] Oct 30 18:23:02.346918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:02.346867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:02.755346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:02.755312 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:03.300532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:03.300502 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c" path="/var/lib/kubelet/pods/c3c1ac4d-9ffa-46a5-a7e9-3a89e15a376c/volumes" Oct 30 18:23:04.779017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:04.778985 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:06.225894 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:23:06.225851 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:23:06.488963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:06.488875 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:08.940202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:08.940170 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:09.748402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:09.748373 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:09.856620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:09.856581 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:10.068848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:10.068756 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb"] Oct 30 18:23:10.070384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:10.070356 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" containerID="cri-o://ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214" gracePeriod=30 Oct 30 18:23:10.090812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:10.090785 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:10.343963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:10.343875 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:10.504603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:10.504571 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:11.530233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:11.530199 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:11.769101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:11.769069 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:11.971672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:11.971642 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh"] Oct 30 18:23:11.976234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:11.976213 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:11.999894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:11.999871 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f42594\"/\"kube-root-ca.crt\"" Oct 30 18:23:12.000046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:11.999942 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f42594\"/\"openshift-service-ca.crt\"" Oct 30 18:23:12.000046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:11.999987 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f42594\"/\"default-dockercfg-7jg8b\"" Oct 30 18:23:12.000046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.000031 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f42594\"/\"ipp-route-f42594-kserve-self-signed-certs\"" Oct 30 18:23:12.002756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.002736 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:12.004059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.004042 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh"] Oct 30 18:23:12.069079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.069046 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-dshm\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.069248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.069175 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-tmp-dir\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.069303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.069258 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jq98\" (UniqueName: \"kubernetes.io/projected/5aa8d637-de56-45b3-aae9-1ce1da124411-kube-api-access-4jq98\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.069303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.069296 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa8d637-de56-45b3-aae9-1ce1da124411-tls-certs\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.069395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.069334 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-home\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.069491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.069472 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-model-cache\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170224 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jq98\" (UniqueName: \"kubernetes.io/projected/5aa8d637-de56-45b3-aae9-1ce1da124411-kube-api-access-4jq98\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170271 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa8d637-de56-45b3-aae9-1ce1da124411-tls-certs\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170315 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-home\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170443 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-model-cache\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170549 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-dshm\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170638 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-tmp-dir\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170811 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-home\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.170968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170913 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-model-cache\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.171037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.170981 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-tmp-dir\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.172913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.172887 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-dshm\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.173226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.173204 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa8d637-de56-45b3-aae9-1ce1da124411-tls-certs\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.191403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.191374 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf"] Oct 30 18:23:12.196595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.196565 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.197755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.197728 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jq98\" (UniqueName: \"kubernetes.io/projected/5aa8d637-de56-45b3-aae9-1ce1da124411-kube-api-access-4jq98\") pod \"ipp-route-f42594-kserve-db64f4f49-w7dwh\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.210505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.210476 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a4f3f0-openshift-default-dockercfg-vqgkj\"" Oct 30 18:23:12.216613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.216579 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf"] Oct 30 18:23:12.238618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.238584 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:12.272389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272353 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-socket\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.272543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272418 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lhj\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-kube-api-access-h6lhj\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.272593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272554 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-certs\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.272636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272610 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-podinfo\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.272683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272671 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istiod-ca-cert\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.272737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272695 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-credential-socket\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.272943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272895 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-envoy\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.273054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.272950 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-token\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.273120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.273055 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-data\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.290519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.290492 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:12.345428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.345391 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374146 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-data\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374211 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-socket\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374291 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lhj\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-kube-api-access-h6lhj\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374380 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-certs\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374440 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-podinfo\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374480 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istiod-ca-cert\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374512 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-credential-socket\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374615 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-envoy\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.374680 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-token\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.375479 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-data\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.375751 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-socket\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.376287 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-certs\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.376974 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-credential-socket\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.380505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.377613 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istiod-ca-cert\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.381660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.380559 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-envoy\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.381660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.381388 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-podinfo\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.403711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.398837 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lhj\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-kube-api-access-h6lhj\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.403711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.400003 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-token\") pod \"e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.538352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.538273 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:12.608259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.607854 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh"] Oct 30 18:23:12.850865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.850837 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf"] Oct 30 18:23:12.853461 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:23:12.853435 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8534d1_561c_4aa7_8380_8bf63ad459e7.slice/crio-ee0e3e3157208ebdceba1c29851f07104f03827ff116e8137167e68d922341f2 WatchSource:0}: Error finding container ee0e3e3157208ebdceba1c29851f07104f03827ff116e8137167e68d922341f2: Status 404 returned error can't find the container with id ee0e3e3157208ebdceba1c29851f07104f03827ff116e8137167e68d922341f2 Oct 30 18:23:12.856209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.856178 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:12.856279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.856244 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:12.856279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.856274 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:12.921303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.921268 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" event={"ID":"5aa8d637-de56-45b3-aae9-1ce1da124411","Type":"ContainerStarted","Data":"c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a"} Oct 30 18:23:12.921466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.921319 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" event={"ID":"5aa8d637-de56-45b3-aae9-1ce1da124411","Type":"ContainerStarted","Data":"c35447f8bdb5f370c15469d31f2f2ebcd85f0697b10949732cf549442939f96c"} Oct 30 18:23:12.923376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.923347 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" event={"ID":"8d8534d1-561c-4aa7-8380-8bf63ad459e7","Type":"ContainerStarted","Data":"ee0e3e3157208ebdceba1c29851f07104f03827ff116e8137167e68d922341f2"} Oct 30 18:23:12.962560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:12.962468 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" podStartSLOduration=1.9624503629999999 podStartE2EDuration="1.962450363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:12.960501917 +0000 UTC m=+1489.784896212" watchObservedRunningTime="2026-10-30 18:23:12.962450363 +0000 UTC m=+1489.786844656" Oct 30 18:23:13.291059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:13.291028 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:13.514176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:13.514140 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:13.932283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:13.932245 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" event={"ID":"8d8534d1-561c-4aa7-8380-8bf63ad459e7","Type":"ContainerStarted","Data":"54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8"} Oct 30 18:23:13.989547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:13.989491 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podStartSLOduration=1.989478944 podStartE2EDuration="1.989478944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:13.976935144 +0000 UTC m=+1490.801329436" watchObservedRunningTime="2026-10-30 18:23:13.989478944 +0000 UTC m=+1490.813873236" Oct 30 18:23:14.253025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:14.252950 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:14.539026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:14.538939 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:14.544508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:14.544481 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:14.939784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:14.939755 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:14.940683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:14.940667 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:23:15.363165 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.363142 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:23:15.525609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525532 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-podinfo\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.525609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525579 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e0fff5-1c1b-4acd-b897-36983b924d27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0fff5-1c1b-4acd-b897-36983b924d27-istiod-ca-cert\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.525863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525652 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-socket\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.525863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525689 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-certs\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.525863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525708 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-token\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-token\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.525863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525758 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-kube-api-access-6wrjm\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-kube-api-access-6wrjm\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.525863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525798 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-envoy\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.526125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525877 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-data\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.526125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.525994 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-credential-socket\") pod \"88e0fff5-1c1b-4acd-b897-36983b924d27\" (UID: \"88e0fff5-1c1b-4acd-b897-36983b924d27\") " Oct 30 18:23:15.526125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526076 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-socket" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:15.526125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526092 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-certs" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:15.526333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526149 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e0fff5-1c1b-4acd-b897-36983b924d27-istiod-ca-cert" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:15.526333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526280 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-data" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:15.526472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526367 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-credential-socket" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:15.526644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526624 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.526743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526643 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e0fff5-1c1b-4acd-b897-36983b924d27-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.526743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526654 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.526743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526666 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.526743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.526679 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.528001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.527970 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-envoy" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:15.528323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.528303 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-podinfo" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:15.528397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.528318 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-token" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:15.528397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.528339 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-kube-api-access-6wrjm" pod "88e0fff5-1c1b-4acd-b897-36983b924d27" (UID: "88e0fff5-1c1b-4acd-b897-36983b924d27"). InnerVolumeSpecName "kube-api-access-6wrjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:15.627719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.627689 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.627719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.627717 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.627913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.627727 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wrjm\" (UniqueName: \"kubernetes.io/projected/88e0fff5-1c1b-4acd-b897-36983b924d27-kube-api-access-6wrjm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.627913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.627737 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e0fff5-1c1b-4acd-b897-36983b924d27-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:15.894132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.894096 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:15.948094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.948062 4582 generic.go:356] "Generic (PLEG): container finished" podID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerID="ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214" exitCode=0 Oct 30 18:23:15.949071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.948121 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" event={"ID":"88e0fff5-1c1b-4acd-b897-36983b924d27","Type":"ContainerDied","Data":"ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214"} Oct 30 18:23:15.949272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.949189 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" event={"ID":"88e0fff5-1c1b-4acd-b897-36983b924d27","Type":"ContainerDied","Data":"cfb850d3d0c7a009e05b791aec272399555457b1d80e119e5a7388238df7bb84"} Oct 30 18:23:15.949272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.949232 4582 scope.go:122] "RemoveContainer" containerID="ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214" Oct 30 18:23:15.949459 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.948212 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb" Oct 30 18:23:15.973494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.973469 4582 scope.go:122] "RemoveContainer" containerID="bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a" Oct 30 18:23:15.984577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.984553 4582 scope.go:122] "RemoveContainer" containerID="ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214" Oct 30 18:23:15.984888 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:23:15.984861 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214\": container with ID starting with ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214 not found: ID does not exist" containerID="ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214" Oct 30 18:23:15.984974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.984893 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214"} err="failed to get container status \"ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214\": rpc error: code = NotFound desc = could not find container \"ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214\": container with ID starting with ec44292d1a4a79d8883ebe147d05e11939190fbdb57b6068c2f1648b8882e214 not found: ID does not exist" Oct 30 18:23:15.984974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.984910 4582 scope.go:122] "RemoveContainer" containerID="bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a" Oct 30 18:23:15.985270 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:23:15.985247 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a\": container with ID starting with bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a not found: ID does not exist" containerID="bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a" Oct 30 18:23:15.985321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:15.985276 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a"} err="failed to get container status \"bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a\": rpc error: code = NotFound desc = could not find container \"bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a\": container with ID starting with bf5a31fd720a6ccfef8a88ec5da177431ebebc06e14ac7b5ba637c91d8e1302a not found: ID does not exist" Oct 30 18:23:16.007076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:16.007043 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb"] Oct 30 18:23:16.016761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:16.016729 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-de14f2-openshift-default-7df9c9ff75-cjbwb"] Oct 30 18:23:16.359388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:16.359358 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:16.472301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:16.472260 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:17.300967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:17.300936 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" path="/var/lib/kubelet/pods/88e0fff5-1c1b-4acd-b897-36983b924d27/volumes" Oct 30 18:23:17.832368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:17.832334 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:18.062033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:18.061996 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:18.159887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:18.159857 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.227586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:20.227548 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.327149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:20.327117 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.440399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:20.440357 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:21.790080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:21.790042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:21.915479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:21.915445 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:22.142697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:22.142671 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:22.291004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:22.290975 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:22.304733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:22.304708 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:23.009440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:23.009413 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:23:23.875168 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:23.875134 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:25.689852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:25.689788 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:27.567989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:27.567954 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:29.271865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:29.271805 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:29.667313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:29.667279 4582 scope.go:122] "RemoveContainer" containerID="1ef21eb78ec86cb83e1ffc98dc2d427858538e5cb242ac693dc96c91e4e23ef7" Oct 30 18:23:29.677714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:29.677693 4582 scope.go:122] "RemoveContainer" containerID="3e3516220afd36753e34a46bb5a454d9db294b97ca7e0f3999e9eff242b9b6b1" Oct 30 18:23:31.162774 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:31.162730 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:33.409061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:33.409027 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:33.660073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:33.659981 4582 patch_prober.go:29] interesting pod/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:23:33.660073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:33.660062 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:35.244282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:35.244246 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:36.114583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:36.114549 4582 generic.go:356] "Generic (PLEG): container finished" podID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerID="83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e" exitCode=0 Oct 30 18:23:36.114789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:36.114623 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" event={"ID":"a0cc9b03-2f67-445f-9e2d-be1a776ee428","Type":"ContainerDied","Data":"83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e"} Oct 30 18:23:36.115065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:36.115048 4582 scope.go:122] "RemoveContainer" containerID="83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e" Oct 30 18:23:36.116649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:36.116619 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:36.116768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:36.116683 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:36.116768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:36.116714 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:36.230955 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:23:36.230918 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:23:37.124067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:37.124036 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" event={"ID":"a0cc9b03-2f67-445f-9e2d-be1a776ee428","Type":"ContainerStarted","Data":"891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89"} Oct 30 18:23:37.200112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:37.200076 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:23:37.200302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:37.200238 4582 patch_prober.go:29] interesting pod/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:23:37.200302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:37.200283 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 18:23:37.502808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:37.502724 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.200149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:38.200116 4582 patch_prober.go:29] interesting pod/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:23:38.200545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:38.200179 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 18:23:39.200321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:39.200286 4582 patch_prober.go:29] interesting pod/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:23:39.200739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:39.200336 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 18:23:39.857844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:39.853227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:40.200187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:40.200151 4582 patch_prober.go:29] interesting pod/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:23:40.200375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:40.200213 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 18:23:41.199472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:41.199437 4582 patch_prober.go:29] interesting pod/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:23:41.199653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:41.199498 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 18:23:41.577781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:41.577699 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:42.241913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:42.241882 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:23:42.243094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:42.243071 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:23:42.244029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:42.244009 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:23:43.769995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:43.769959 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:43.874578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:43.874544 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:43.979029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:43.978988 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:44.868020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:44.867987 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:45.801077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.801042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:45.826272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.826241 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq"] Oct 30 18:23:45.826879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.826862 4582 cpu_manager.go:414] "containerMap: removing container" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" Oct 30 18:23:45.826953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.826881 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" Oct 30 18:23:45.826953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.826911 4582 cpu_manager.go:414] "containerMap: removing container" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" Oct 30 18:23:45.826953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.826917 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" Oct 30 18:23:45.827107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.827050 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" Oct 30 18:23:45.827107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.827067 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e0fff5-1c1b-4acd-b897-36983b924d27" containerName="istio-proxy" Oct 30 18:23:45.829773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.829752 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:45.843954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.843924 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:23:45.854046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.854017 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:23:45.854046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.854028 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:23:45.854249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.854059 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:23:45.859918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:45.859895 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq"] Oct 30 18:23:46.005475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.005440 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.005907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.005551 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.005907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.005666 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjx59\" (UniqueName: \"kubernetes.io/projected/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-kube-api-access-wjx59\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.107269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.107190 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.107416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.107306 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjx59\" (UniqueName: \"kubernetes.io/projected/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-kube-api-access-wjx59\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.107465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.107433 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.107898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.107872 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.109740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.109718 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.123637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.123609 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjx59\" (UniqueName: \"kubernetes.io/projected/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-kube-api-access-wjx59\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.142908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.142885 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:46.440251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.440225 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq"] Oct 30 18:23:46.680768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.680729 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:46.781816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:46.781782 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:47.233975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:47.233940 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" event={"ID":"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4","Type":"ContainerStarted","Data":"f1de7658a4d2933a1b45ee59717bcc979c2d9e5637cb48b2c90659e32dfb064e"} Oct 30 18:23:47.234387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:47.233997 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" event={"ID":"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4","Type":"ContainerStarted","Data":"9ac9ac7152822757d27469f82c173ce95a05724a09e20005500e4738d2230b85"} Oct 30 18:23:47.269523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:47.269469 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" podStartSLOduration=2.269453061 podStartE2EDuration="2.269453061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:47.267295034 +0000 UTC m=+1524.091689346" watchObservedRunningTime="2026-10-30 18:23:47.269453061 +0000 UTC m=+1524.093847353" Oct 30 18:23:48.029593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:48.029555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:48.241715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:48.241685 4582 generic.go:356] "Generic (PLEG): container finished" podID="b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" containerID="f1de7658a4d2933a1b45ee59717bcc979c2d9e5637cb48b2c90659e32dfb064e" exitCode=0 Oct 30 18:23:48.242121 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:48.241771 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" event={"ID":"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4","Type":"ContainerDied","Data":"f1de7658a4d2933a1b45ee59717bcc979c2d9e5637cb48b2c90659e32dfb064e"} Oct 30 18:23:49.026798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.026768 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl"] Oct 30 18:23:49.031371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.031341 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d"] Oct 30 18:23:49.035914 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.035891 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73668924-lpbsl"] Oct 30 18:23:49.040363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.040336 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341e8382-ddw6d"] Oct 30 18:23:49.301552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.301482 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0a411c-af6b-418a-b128-917efdf2c602" path="/var/lib/kubelet/pods/bc0a411c-af6b-418a-b128-917efdf2c602/volumes" Oct 30 18:23:49.302073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.302040 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7625e37-12c0-4366-950a-ae4ab7fa4350" path="/var/lib/kubelet/pods/f7625e37-12c0-4366-950a-ae4ab7fa4350/volumes" Oct 30 18:23:49.476649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.476617 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:49.587523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.587441 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-kube-api-access-wjx59\" (UniqueName: \"kubernetes.io/projected/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-kube-api-access-wjx59\") pod \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " Oct 30 18:23:49.587523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.587508 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-xks-ca\") pod \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " Oct 30 18:23:49.587712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.587634 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-openshift-ca\") pod \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\" (UID: \"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4\") " Oct 30 18:23:49.588059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.588033 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-openshift-ca" pod "b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" (UID: "b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:49.588302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.588285 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.589602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.589566 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-kube-api-access-wjx59" pod "b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" (UID: "b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4"). InnerVolumeSpecName "kube-api-access-wjx59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:49.589720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.589635 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-xks-ca" pod "b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" (UID: "b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:49.689418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.689383 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjx59\" (UniqueName: \"kubernetes.io/projected/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-kube-api-access-wjx59\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.689418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:49.689415 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:50.280484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:50.280453 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" Oct 30 18:23:50.280689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:50.280460 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq" event={"ID":"b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4","Type":"ContainerDied","Data":"9ac9ac7152822757d27469f82c173ce95a05724a09e20005500e4738d2230b85"} Oct 30 18:23:50.280689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:50.280606 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ac9ac7152822757d27469f82c173ce95a05724a09e20005500e4738d2230b85" Oct 30 18:23:50.368351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:50.368318 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:52.396967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:52.396933 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:53.572842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:53.571427 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:53.667346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:53.667313 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:54.201881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:54.201346 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:55.878882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.878812 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq"] Oct 30 18:23:55.879813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.879795 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" containerName="revoke-keys" Oct 30 18:23:55.879893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.879815 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" containerName="revoke-keys" Oct 30 18:23:55.880000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.879988 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" containerName="revoke-keys" Oct 30 18:23:55.883215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.883195 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:55.899619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.899595 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq"] Oct 30 18:23:55.905230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.905197 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:23:55.905230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.905216 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:23:55.905381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.905216 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:23:55.905381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:55.905287 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:23:56.012685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.012636 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5h5\" (UniqueName: \"kubernetes.io/projected/b5645b94-e959-4c32-861c-2adea9ebbbca-kube-api-access-zb5h5\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.012913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.012788 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.013095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.013071 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.114494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.114453 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.114687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.114533 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5h5\" (UniqueName: \"kubernetes.io/projected/b5645b94-e959-4c32-861c-2adea9ebbbca-kube-api-access-zb5h5\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.114687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.114636 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.115196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.115168 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.117050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.117022 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.131239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.131180 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5h5\" (UniqueName: \"kubernetes.io/projected/b5645b94-e959-4c32-861c-2adea9ebbbca-kube-api-access-zb5h5\") pod \"maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.146364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.146338 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:56.197514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.197487 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:56.525529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:56.525500 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq"] Oct 30 18:23:56.528329 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:23:56.528301 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5645b94_e959_4c32_861c_2adea9ebbbca.slice/crio-e6883a002b92121af9cde3aef5ef96373ec0c9445103f095e45cc0d4a7d0c2dd WatchSource:0}: Error finding container e6883a002b92121af9cde3aef5ef96373ec0c9445103f095e45cc0d4a7d0c2dd: Status 404 returned error can't find the container with id e6883a002b92121af9cde3aef5ef96373ec0c9445103f095e45cc0d4a7d0c2dd Oct 30 18:23:57.339243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:57.339210 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" event={"ID":"b5645b94-e959-4c32-861c-2adea9ebbbca","Type":"ContainerStarted","Data":"bb8b7a969e63764d97d2bb2321df61b5c342fb0b82c0d2bbdcc27171e7335320"} Oct 30 18:23:57.339757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:57.339266 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" event={"ID":"b5645b94-e959-4c32-861c-2adea9ebbbca","Type":"ContainerStarted","Data":"e6883a002b92121af9cde3aef5ef96373ec0c9445103f095e45cc0d4a7d0c2dd"} Oct 30 18:23:57.374539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:57.374482 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" podStartSLOduration=2.374468263 podStartE2EDuration="2.374468263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:57.370411159 +0000 UTC m=+1534.194805450" watchObservedRunningTime="2026-10-30 18:23:57.374468263 +0000 UTC m=+1534.198862555" Oct 30 18:23:57.946988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:57.946960 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:58.347449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.347350 4582 generic.go:356] "Generic (PLEG): container finished" podID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerID="9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08" exitCode=0 Oct 30 18:23:58.347449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.347431 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" event={"ID":"f10a41fd-276b-4bf5-879d-300ef0977e38","Type":"ContainerDied","Data":"9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08"} Oct 30 18:23:58.348175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.347865 4582 scope.go:122] "RemoveContainer" containerID="9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08" Oct 30 18:23:58.349323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.349303 4582 generic.go:356] "Generic (PLEG): container finished" podID="b5645b94-e959-4c32-861c-2adea9ebbbca" containerID="bb8b7a969e63764d97d2bb2321df61b5c342fb0b82c0d2bbdcc27171e7335320" exitCode=0 Oct 30 18:23:58.349435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.349388 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" event={"ID":"b5645b94-e959-4c32-861c-2adea9ebbbca","Type":"ContainerDied","Data":"bb8b7a969e63764d97d2bb2321df61b5c342fb0b82c0d2bbdcc27171e7335320"} Oct 30 18:23:58.349511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.349484 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:58.349589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.349573 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:58.349664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:58.349649 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:23:59.367170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.364142 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" event={"ID":"f10a41fd-276b-4bf5-879d-300ef0977e38","Type":"ContainerStarted","Data":"42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8"} Oct 30 18:23:59.565847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.547650 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:23:59.567099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.567063 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:23:59.567321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.567296 4582 patch_prober.go:29] interesting pod/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:23:59.567395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.567345 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 18:23:59.574836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.574796 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-openshift-ca\") pod \"b5645b94-e959-4c32-861c-2adea9ebbbca\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " Oct 30 18:23:59.574961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.574859 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-xks-ca\") pod \"b5645b94-e959-4c32-861c-2adea9ebbbca\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " Oct 30 18:23:59.575025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.574975 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5645b94-e959-4c32-861c-2adea9ebbbca-kube-api-access-zb5h5\" (UniqueName: \"kubernetes.io/projected/b5645b94-e959-4c32-861c-2adea9ebbbca-kube-api-access-zb5h5\") pod \"b5645b94-e959-4c32-861c-2adea9ebbbca\" (UID: \"b5645b94-e959-4c32-861c-2adea9ebbbca\") " Oct 30 18:23:59.575231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.575204 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-openshift-ca" pod "b5645b94-e959-4c32-861c-2adea9ebbbca" (UID: "b5645b94-e959-4c32-861c-2adea9ebbbca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:59.575554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.575537 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:59.576987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.576962 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-xks-ca" pod "b5645b94-e959-4c32-861c-2adea9ebbbca" (UID: "b5645b94-e959-4c32-861c-2adea9ebbbca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:59.577062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.576989 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5645b94-e959-4c32-861c-2adea9ebbbca-kube-api-access-zb5h5" pod "b5645b94-e959-4c32-861c-2adea9ebbbca" (UID: "b5645b94-e959-4c32-861c-2adea9ebbbca"). InnerVolumeSpecName "kube-api-access-zb5h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:59.677356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.677312 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5645b94-e959-4c32-861c-2adea9ebbbca-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:59.677356 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.677343 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5h5\" (UniqueName: \"kubernetes.io/projected/b5645b94-e959-4c32-861c-2adea9ebbbca-kube-api-access-zb5h5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:23:59.684900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:23:59.684870 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:00.374988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:00.374954 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" Oct 30 18:24:00.375381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:00.374997 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq" event={"ID":"b5645b94-e959-4c32-861c-2adea9ebbbca","Type":"ContainerDied","Data":"e6883a002b92121af9cde3aef5ef96373ec0c9445103f095e45cc0d4a7d0c2dd"} Oct 30 18:24:00.375381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:00.375049 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6883a002b92121af9cde3aef5ef96373ec0c9445103f095e45cc0d4a7d0c2dd" Oct 30 18:24:00.558716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:00.558243 4582 patch_prober.go:29] interesting pod/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:24:00.558716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:00.558310 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 18:24:01.533963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:01.533925 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:01.567225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:01.567185 4582 patch_prober.go:29] interesting pod/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:24:01.567429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:01.567249 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 18:24:02.556329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:02.556293 4582 patch_prober.go:29] interesting pod/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:24:02.556965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:02.556934 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 18:24:03.562280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:03.562241 4582 patch_prober.go:29] interesting pod/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:24:03.562911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:03.562877 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 18:24:03.978734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:03.978691 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:04.562310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:04.562264 4582 patch_prober.go:29] interesting pod/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 18:24:04.562983 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:04.562948 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 18:24:05.570651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:05.570617 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:24:05.571109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:05.571086 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:24:05.571802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:05.571774 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:24:05.775154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:05.775119 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:05.867905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:05.867868 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:06.225789 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:24:06.225713 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:24:06.671335 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:06.671300 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:06.779292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:06.779254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:08.213782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:08.213744 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:10.611635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:10.611592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:12.184468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:12.184433 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:12.394319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:12.394288 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:12.493257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:12.493182 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:12.941805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:12.941763 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:13.100275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.100230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:13.235397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.235321 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:13.313959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.313925 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq"] Oct 30 18:24:13.322053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.322022 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9096a-rjdtq"] Oct 30 18:24:13.415014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.414964 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:13.512624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.512528 4582 generic.go:356] "Generic (PLEG): container finished" podID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerID="54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8" exitCode=0 Oct 30 18:24:13.512624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.512594 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" event={"ID":"8d8534d1-561c-4aa7-8380-8bf63ad459e7","Type":"ContainerDied","Data":"54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8"} Oct 30 18:24:13.513258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.513239 4582 scope.go:122] "RemoveContainer" containerID="54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8" Oct 30 18:24:13.515569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.515536 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:13.515688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.515607 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:13.515688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.515644 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:13.524547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.524521 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:13.758975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:13.758940 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:14.367761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:14.367718 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:14.522724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:14.522693 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" event={"ID":"8d8534d1-561c-4aa7-8380-8bf63ad459e7","Type":"ContainerStarted","Data":"fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639"} Oct 30 18:24:14.538868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:14.538845 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:24:14.539796 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:14.539771 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:24:14.539937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:14.539853 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 18:24:14.945991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:14.945948 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:24:15.359218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:15.346423 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4" path="/var/lib/kubelet/pods/b0ea9b6c-8791-4f1e-829f-5a2e21b7abf4/volumes" Oct 30 18:24:15.539106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:15.539077 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:24:15.539483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:15.539124 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 18:24:15.868196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:15.868163 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:16.538835 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:16.538800 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:24:16.539004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:16.538870 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 18:24:17.538593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:17.538556 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:24:17.539103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:17.538625 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 18:24:18.320956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:18.320905 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:18.539365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:18.539335 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:24:18.539740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:18.539383 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 18:24:19.539574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:19.539541 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:24:19.540015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:19.539596 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 18:24:19.733675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:19.733643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:19.905483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:19.905441 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:20.295677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:20.295585 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:20.383083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:20.383042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:20.544249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:20.544218 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:24:20.545200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:20.545183 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:24:20.605416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:20.605332 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:20.702004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:20.701966 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:21.126798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.126766 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:21.240303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.240270 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg"] Oct 30 18:24:21.241070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.241043 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b5645b94-e959-4c32-861c-2adea9ebbbca" containerName="revoke-keys" Oct 30 18:24:21.241070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.241068 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5645b94-e959-4c32-861c-2adea9ebbbca" containerName="revoke-keys" Oct 30 18:24:21.241269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.241164 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5645b94-e959-4c32-861c-2adea9ebbbca" containerName="revoke-keys" Oct 30 18:24:21.244177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.244156 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.264283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.264253 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg"] Oct 30 18:24:21.268961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.268938 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:24:21.268961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.268938 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:21.269115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.268944 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:24:21.269115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.268982 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:24:21.338646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.338605 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmcfp\" (UniqueName: \"kubernetes.io/projected/1df38516-8aca-40ad-99e3-e777355be7f6-kube-api-access-wmcfp\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.338853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.338729 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.338853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.338814 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.440316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.440231 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.440316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.440288 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.440520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.440494 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmcfp\" (UniqueName: \"kubernetes.io/projected/1df38516-8aca-40ad-99e3-e777355be7f6-kube-api-access-wmcfp\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.440942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.440918 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.442595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.442574 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.457276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.457253 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmcfp\" (UniqueName: \"kubernetes.io/projected/1df38516-8aca-40ad-99e3-e777355be7f6-kube-api-access-wmcfp\") pod \"maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.557589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.557562 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:21.859082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:21.859050 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg"] Oct 30 18:24:21.862120 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:24:21.862094 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df38516_8aca_40ad_99e3_e777355be7f6.slice/crio-cd1d6f17f372ebd35fc1cd87e646065d815e10ae69b02b099216c36e71d6a688 WatchSource:0}: Error finding container cd1d6f17f372ebd35fc1cd87e646065d815e10ae69b02b099216c36e71d6a688: Status 404 returned error can't find the container with id cd1d6f17f372ebd35fc1cd87e646065d815e10ae69b02b099216c36e71d6a688 Oct 30 18:24:22.607077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:22.606984 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" event={"ID":"1df38516-8aca-40ad-99e3-e777355be7f6","Type":"ContainerStarted","Data":"4ae752122768366e6e98a02799c79c0cf41ad83171d71e6c9e45dd6314c68589"} Oct 30 18:24:22.607077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:22.607055 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" event={"ID":"1df38516-8aca-40ad-99e3-e777355be7f6","Type":"ContainerStarted","Data":"cd1d6f17f372ebd35fc1cd87e646065d815e10ae69b02b099216c36e71d6a688"} Oct 30 18:24:22.643720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:22.643652 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" podStartSLOduration=1.643635146 podStartE2EDuration="1.643635146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:22.640863161 +0000 UTC m=+1559.465257457" watchObservedRunningTime="2026-10-30 18:24:22.643635146 +0000 UTC m=+1559.468029438" Oct 30 18:24:23.058400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:23.058367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:23.615736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:23.615702 4582 generic.go:356] "Generic (PLEG): container finished" podID="1df38516-8aca-40ad-99e3-e777355be7f6" containerID="4ae752122768366e6e98a02799c79c0cf41ad83171d71e6c9e45dd6314c68589" exitCode=0 Oct 30 18:24:23.616176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:23.615791 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" event={"ID":"1df38516-8aca-40ad-99e3-e777355be7f6","Type":"ContainerDied","Data":"4ae752122768366e6e98a02799c79c0cf41ad83171d71e6c9e45dd6314c68589"} Oct 30 18:24:24.713855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.713779 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:24.836453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.836424 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:24.885425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.885345 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-xks-ca\") pod \"1df38516-8aca-40ad-99e3-e777355be7f6\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " Oct 30 18:24:24.885599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.885539 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df38516-8aca-40ad-99e3-e777355be7f6-kube-api-access-wmcfp\" (UniqueName: \"kubernetes.io/projected/1df38516-8aca-40ad-99e3-e777355be7f6-kube-api-access-wmcfp\") pod \"1df38516-8aca-40ad-99e3-e777355be7f6\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " Oct 30 18:24:24.885686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.885669 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-openshift-ca\") pod \"1df38516-8aca-40ad-99e3-e777355be7f6\" (UID: \"1df38516-8aca-40ad-99e3-e777355be7f6\") " Oct 30 18:24:24.886107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.886080 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-openshift-ca" pod "1df38516-8aca-40ad-99e3-e777355be7f6" (UID: "1df38516-8aca-40ad-99e3-e777355be7f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:24.886507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.886483 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:24.887610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.887575 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-xks-ca" pod "1df38516-8aca-40ad-99e3-e777355be7f6" (UID: "1df38516-8aca-40ad-99e3-e777355be7f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:24.887784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.887763 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df38516-8aca-40ad-99e3-e777355be7f6-kube-api-access-wmcfp" pod "1df38516-8aca-40ad-99e3-e777355be7f6" (UID: "1df38516-8aca-40ad-99e3-e777355be7f6"). InnerVolumeSpecName "kube-api-access-wmcfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:24.992853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.992781 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df38516-8aca-40ad-99e3-e777355be7f6-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:24.992853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:24.992831 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmcfp\" (UniqueName: \"kubernetes.io/projected/1df38516-8aca-40ad-99e3-e777355be7f6-kube-api-access-wmcfp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:25.629610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:25.629584 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" Oct 30 18:24:25.629772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:25.629631 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg" event={"ID":"1df38516-8aca-40ad-99e3-e777355be7f6","Type":"ContainerDied","Data":"cd1d6f17f372ebd35fc1cd87e646065d815e10ae69b02b099216c36e71d6a688"} Oct 30 18:24:25.629772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:25.629674 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd1d6f17f372ebd35fc1cd87e646065d815e10ae69b02b099216c36e71d6a688" Oct 30 18:24:26.437869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:26.437788 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:28.210972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:28.210940 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:29.713536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:29.713500 4582 scope.go:122] "RemoveContainer" containerID="978e8c8d790aec07ec9b6a20cbe1cdb744876e130d14906d97ec5010509862e8" Oct 30 18:24:29.724371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:29.724346 4582 scope.go:122] "RemoveContainer" containerID="32ea10073df8e839b23ea4fa2e54aa0ee3ed2907c76a8ad662adcd2aa3b953f7" Oct 30 18:24:29.733405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:29.733387 4582 scope.go:122] "RemoveContainer" containerID="f2a84763d56d1dc61f7dc2d89b6b7060b086e967ecb2589d8a8ce511d1674fdd" Oct 30 18:24:29.742001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:29.741977 4582 scope.go:122] "RemoveContainer" containerID="2faec04c0db4d3b820911bd7849f8e9f1f0b4a434fe6eb376465bb403535bd15" Oct 30 18:24:30.090565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:30.090492 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:31.809071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:31.809038 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.232906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.232875 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv"] Oct 30 18:24:32.235768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.233608 4582 cpu_manager.go:414] "containerMap: removing container" podUID="1df38516-8aca-40ad-99e3-e777355be7f6" containerName="revoke-keys" Oct 30 18:24:32.235768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.233629 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df38516-8aca-40ad-99e3-e777355be7f6" containerName="revoke-keys" Oct 30 18:24:32.235768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.233758 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df38516-8aca-40ad-99e3-e777355be7f6" containerName="revoke-keys" Oct 30 18:24:32.238628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.238606 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.253192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.253171 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-200c58-openshift-default-dockercfg-wszbc\"" Oct 30 18:24:32.267482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.267457 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv"] Oct 30 18:24:32.289475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.289445 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.391898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.391855 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-credential-socket\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.391945 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-data\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.391989 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-podinfo\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.392118 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-envoy\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.392247 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istiod-ca-cert\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.392286 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-socket\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.392336 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-token\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.392405 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-certs\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.392528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.392463 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z77sr\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-kube-api-access-z77sr\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.476408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.476372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.496606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.496513 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istiod-ca-cert\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.497704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.497681 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-socket\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.498237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.498218 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-token\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.498467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.498449 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-certs\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.498651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.498635 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z77sr\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-kube-api-access-z77sr\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.498936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.498917 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-credential-socket\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.499123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.499106 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-data\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.499270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.499252 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-podinfo\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.499417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.499403 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-envoy\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.500247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.500226 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-certs\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.500405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.497599 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istiod-ca-cert\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.500848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.498150 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-socket\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.501240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.501206 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-credential-socket\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.501943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.501924 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-data\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.505259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.505238 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-podinfo\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.505907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.505886 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-envoy\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.531775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.531744 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-token\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.532410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.532384 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z77sr\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-kube-api-access-z77sr\") pod \"e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.553380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.553345 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:32.584552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.584524 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:32.871595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.871572 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv"] Oct 30 18:24:32.874730 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:24:32.874704 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fae2ec_2b00_45dd_b9ef_06decebd68b9.slice/crio-b2a48696d27e4675cc96c5aa0fbb0a6cd86c260bbf08d6dd26ea7dc3ff5986e8 WatchSource:0}: Error finding container b2a48696d27e4675cc96c5aa0fbb0a6cd86c260bbf08d6dd26ea7dc3ff5986e8: Status 404 returned error can't find the container with id b2a48696d27e4675cc96c5aa0fbb0a6cd86c260bbf08d6dd26ea7dc3ff5986e8 Oct 30 18:24:32.877859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.877798 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:32.877956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.877894 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:32.877956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:32.877943 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:33.688438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:33.688400 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" event={"ID":"94fae2ec-2b00-45dd-b9ef-06decebd68b9","Type":"ContainerStarted","Data":"63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2"} Oct 30 18:24:33.688611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:33.688452 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" event={"ID":"94fae2ec-2b00-45dd-b9ef-06decebd68b9","Type":"ContainerStarted","Data":"b2a48696d27e4675cc96c5aa0fbb0a6cd86c260bbf08d6dd26ea7dc3ff5986e8"} Oct 30 18:24:33.736533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:33.736487 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" podStartSLOduration=1.736475695 podStartE2EDuration="1.736475695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:33.735230736 +0000 UTC m=+1570.559625028" watchObservedRunningTime="2026-10-30 18:24:33.736475695 +0000 UTC m=+1570.560869986" Oct 30 18:24:34.504474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:34.504441 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:34.553793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:34.553761 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:34.559784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:34.559758 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:34.695672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:34.695644 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:34.696513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:34.696493 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:24:35.829239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.829206 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm"] Oct 30 18:24:35.832570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.832553 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:35.850588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.850562 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm"] Oct 30 18:24:35.855940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.855913 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:24:35.855940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.855925 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:35.856084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.856055 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:24:35.857114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.857096 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:24:35.951880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.951845 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:35.952069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.952012 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:35.952129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:35.952072 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfzw\" (UniqueName: \"kubernetes.io/projected/0e159a0c-4e6b-4243-b880-6ab7f1b77212-kube-api-access-pxfzw\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.053618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.053566 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.053857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.053691 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.053857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.053758 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfzw\" (UniqueName: \"kubernetes.io/projected/0e159a0c-4e6b-4243-b880-6ab7f1b77212-kube-api-access-pxfzw\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.054347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.054324 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.055915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.055893 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.070252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.070227 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfzw\" (UniqueName: \"kubernetes.io/projected/0e159a0c-4e6b-4243-b880-6ab7f1b77212-kube-api-access-pxfzw\") pod \"maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.103347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.103284 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:36.145709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.145674 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:36.206037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.205989 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:36.245996 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:24:36.245955 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:24:36.318252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.318219 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:36.446014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.445981 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm"] Oct 30 18:24:36.449292 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:24:36.449257 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e159a0c_4e6b_4243_b880_6ab7f1b77212.slice/crio-8b7d2e4734be8b984b85bcf0ae8b2a8512dfd66e8415df496f5155d3f6bf7980 WatchSource:0}: Error finding container 8b7d2e4734be8b984b85bcf0ae8b2a8512dfd66e8415df496f5155d3f6bf7980: Status 404 returned error can't find the container with id 8b7d2e4734be8b984b85bcf0ae8b2a8512dfd66e8415df496f5155d3f6bf7980 Oct 30 18:24:36.683986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.683947 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:36.715736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.715650 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" event={"ID":"0e159a0c-4e6b-4243-b880-6ab7f1b77212","Type":"ContainerStarted","Data":"8b7d2e4734be8b984b85bcf0ae8b2a8512dfd66e8415df496f5155d3f6bf7980"} Oct 30 18:24:36.889720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.889682 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:36.996222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:36.996148 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:37.567885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:37.567839 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:37.680245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:37.680213 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:37.724537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:37.724451 4582 generic.go:356] "Generic (PLEG): container finished" podID="0e159a0c-4e6b-4243-b880-6ab7f1b77212" containerID="5a871d6ed599e42c11cb2211bd84e2c12c961e2324e1f2f7945dcd8e425a3131" exitCode=0 Oct 30 18:24:37.724711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:37.724535 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" event={"ID":"0e159a0c-4e6b-4243-b880-6ab7f1b77212","Type":"ContainerDied","Data":"5a871d6ed599e42c11cb2211bd84e2c12c961e2324e1f2f7945dcd8e425a3131"} Oct 30 18:24:37.755794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:37.755759 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg"] Oct 30 18:24:37.760491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:37.760459 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-76135e-2xcbg"] Oct 30 18:24:37.791752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:37.791721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:38.198895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.198854 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:38.278727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.278697 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p"] Oct 30 18:24:38.278939 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.278896 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" containerID="cri-o://cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb" gracePeriod=30 Oct 30 18:24:38.287367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.287345 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl"] Oct 30 18:24:38.287542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.287522 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" containerID="cri-o://b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1" gracePeriod=30 Oct 30 18:24:38.291672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.291647 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx"] Oct 30 18:24:38.291938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.291835 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" containerID="cri-o://33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90" gracePeriod=30 Oct 30 18:24:38.448655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.448621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:38.542299 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:38.542221 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:39.060709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.060685 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:39.125814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.125782 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-xks-ca\") pod \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " Oct 30 18:24:39.126009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.125871 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-openshift-ca\") pod \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " Oct 30 18:24:39.126009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.125938 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e159a0c-4e6b-4243-b880-6ab7f1b77212-kube-api-access-pxfzw\" (UniqueName: \"kubernetes.io/projected/0e159a0c-4e6b-4243-b880-6ab7f1b77212-kube-api-access-pxfzw\") pod \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\" (UID: \"0e159a0c-4e6b-4243-b880-6ab7f1b77212\") " Oct 30 18:24:39.126256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.126227 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-openshift-ca" pod "0e159a0c-4e6b-4243-b880-6ab7f1b77212" (UID: "0e159a0c-4e6b-4243-b880-6ab7f1b77212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:39.126631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.126612 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.128042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.128013 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e159a0c-4e6b-4243-b880-6ab7f1b77212-kube-api-access-pxfzw" pod "0e159a0c-4e6b-4243-b880-6ab7f1b77212" (UID: "0e159a0c-4e6b-4243-b880-6ab7f1b77212"). InnerVolumeSpecName "kube-api-access-pxfzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:39.128144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.128045 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-xks-ca" pod "0e159a0c-4e6b-4243-b880-6ab7f1b77212" (UID: "0e159a0c-4e6b-4243-b880-6ab7f1b77212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:39.227472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.227420 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxfzw\" (UniqueName: \"kubernetes.io/projected/0e159a0c-4e6b-4243-b880-6ab7f1b77212-kube-api-access-pxfzw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.227472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.227442 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e159a0c-4e6b-4243-b880-6ab7f1b77212-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.300981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.300948 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df38516-8aca-40ad-99e3-e777355be7f6" path="/var/lib/kubelet/pods/1df38516-8aca-40ad-99e3-e777355be7f6/volumes" Oct 30 18:24:39.741995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.741962 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" event={"ID":"0e159a0c-4e6b-4243-b880-6ab7f1b77212","Type":"ContainerDied","Data":"8b7d2e4734be8b984b85bcf0ae8b2a8512dfd66e8415df496f5155d3f6bf7980"} Oct 30 18:24:39.741995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.741992 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm" Oct 30 18:24:39.742215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.742043 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7d2e4734be8b984b85bcf0ae8b2a8512dfd66e8415df496f5155d3f6bf7980" Oct 30 18:24:39.872955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:39.872922 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:41.587462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:41.587433 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.221013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:42.220983 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.340122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:42.340088 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.389792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:42.389712 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq"] Oct 30 18:24:42.405686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:42.405650 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10f1a3-fvssq"] Oct 30 18:24:42.562369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:42.562295 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.853904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:42.853795 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:43.305864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.305815 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5645b94-e959-4c32-861c-2adea9ebbbca" path="/var/lib/kubelet/pods/b5645b94-e959-4c32-861c-2adea9ebbbca/volumes" Oct 30 18:24:43.590475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.590446 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:24:43.627075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.627054 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:24:43.660057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.660024 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:24:43.711143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711109 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-podinfo\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.711318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711164 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-token\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-token\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.711318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711200 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-certs\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.711318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711234 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-data\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.711318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711262 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-data\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.711553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711420 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-podinfo\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.711553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711484 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-certs\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.711553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711490 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-data" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.711553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711543 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-kube-api-access-nz7km\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-kube-api-access-nz7km\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.711779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711579 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4802e94-077b-4c9b-a036-44903c74f610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4802e94-077b-4c9b-a036-44903c74f610-istio-podinfo\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.711779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711624 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istiod-ca-cert\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.711779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711668 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-credential-socket\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.711779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711695 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-socket\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.711779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711728 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-credential-socket\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.711779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711776 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-credential-socket\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.712067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711803 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-istio-token\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-istio-token\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.712067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711967 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-certs\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.712067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712004 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4802e94-077b-4c9b-a036-44903c74f610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4802e94-077b-4c9b-a036-44903c74f610-istiod-ca-cert\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.712067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712052 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-socket\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.712197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712083 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-token\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-token\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.712197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712113 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-kube-api-access-lfz6j\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-kube-api-access-lfz6j\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.712197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712148 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-envoy\") pod \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\" (UID: \"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8\") " Oct 30 18:24:43.712197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712186 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-kube-api-access-c4jg7\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-kube-api-access-c4jg7\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.712344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712266 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-data\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.712344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712341 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-envoy\") pod \"f4802e94-077b-4c9b-a036-44903c74f610\" (UID: \"f4802e94-077b-4c9b-a036-44903c74f610\") " Oct 30 18:24:43.712466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712386 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-socket\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.712502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712462 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a90fe9e-b661-483d-a375-d5bfe196be63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a90fe9e-b661-483d-a375-d5bfe196be63-istiod-ca-cert\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.712502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712490 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-envoy\") pod \"6a90fe9e-b661-483d-a375-d5bfe196be63\" (UID: \"6a90fe9e-b661-483d-a375-d5bfe196be63\") " Oct 30 18:24:43.713334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.713311 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.716345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.711728 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-certs" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712058 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-certs" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.712115 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istiod-ca-cert" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.713925 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-podinfo" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.713965 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-socket" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.714162 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-data" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.714420 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-credential-socket" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.714641 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-socket" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.714832 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-credential-socket" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.715013 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-certs" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.715220 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-socket" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.715329 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4802e94-077b-4c9b-a036-44903c74f610-istiod-ca-cert" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.716510 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-credential-socket" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.715434 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-envoy" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.715459 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a90fe9e-b661-483d-a375-d5bfe196be63-istiod-ca-cert" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.715727 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-data" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.715783 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-podinfo" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:43.716719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.716317 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-kube-api-access-lfz6j" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "kube-api-access-lfz6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.718707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.718683 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-envoy" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.720343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.720317 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-envoy" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.720449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.720401 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-istio-token" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.722208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.722136 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4802e94-077b-4c9b-a036-44903c74f610-istio-podinfo" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:43.722208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.722177 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-kube-api-access-nz7km" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "kube-api-access-nz7km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.722409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.722202 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-token" pod "6a90fe9e-b661-483d-a375-d5bfe196be63" (UID: "6a90fe9e-b661-483d-a375-d5bfe196be63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.731428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.725018 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-token" pod "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" (UID: "f8a665a0-c0d5-4c83-bee2-89706cfa8ad8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.731428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.730938 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-kube-api-access-c4jg7" pod "f4802e94-077b-4c9b-a036-44903c74f610" (UID: "f4802e94-077b-4c9b-a036-44903c74f610"). InnerVolumeSpecName "kube-api-access-c4jg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.781437 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.781414 4582 generic.go:356] "Generic (PLEG): container finished" podID="f4802e94-077b-4c9b-a036-44903c74f610" containerID="33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90" exitCode=0 Oct 30 18:24:43.781548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.781455 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" Oct 30 18:24:43.781548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.781502 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" event={"ID":"f4802e94-077b-4c9b-a036-44903c74f610","Type":"ContainerDied","Data":"33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90"} Oct 30 18:24:43.781666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.781550 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx" event={"ID":"f4802e94-077b-4c9b-a036-44903c74f610","Type":"ContainerDied","Data":"aaccff52d6d1c76cb22236aab082886cf89909a4dfebeae08e0ab5c5f0a4da78"} Oct 30 18:24:43.781666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.781575 4582 scope.go:122] "RemoveContainer" containerID="33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90" Oct 30 18:24:43.783157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.783135 4582 generic.go:356] "Generic (PLEG): container finished" podID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerID="b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1" exitCode=0 Oct 30 18:24:43.783251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.783191 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" Oct 30 18:24:43.783251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.783227 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" event={"ID":"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8","Type":"ContainerDied","Data":"b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1"} Oct 30 18:24:43.783354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.783274 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl" event={"ID":"f8a665a0-c0d5-4c83-bee2-89706cfa8ad8","Type":"ContainerDied","Data":"00536971aeff31fd5c387e008e6500bc239c3cb2a0f7d3be679f06ab8ffc9945"} Oct 30 18:24:43.785015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.784993 4582 generic.go:356] "Generic (PLEG): container finished" podID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerID="cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb" exitCode=0 Oct 30 18:24:43.785120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.785037 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" Oct 30 18:24:43.785182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.785033 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" event={"ID":"6a90fe9e-b661-483d-a375-d5bfe196be63","Type":"ContainerDied","Data":"cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb"} Oct 30 18:24:43.785182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.785171 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p" event={"ID":"6a90fe9e-b661-483d-a375-d5bfe196be63","Type":"ContainerDied","Data":"6ae918540135a19efe4cb8861ae76e33466e818b3dc707ee42c5efc426e8ef47"} Oct 30 18:24:43.797268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.797246 4582 scope.go:122] "RemoveContainer" containerID="33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90" Oct 30 18:24:43.798071 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:24:43.798040 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90\": container with ID starting with 33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90 not found: ID does not exist" containerID="33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90" Oct 30 18:24:43.798168 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.798080 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90"} err="failed to get container status \"33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90\": rpc error: code = NotFound desc = could not find container \"33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90\": container with ID starting with 33866e1a105f43374109711ee69efa08e74546f8da5605d552511b7c7380df90 not found: ID does not exist" Oct 30 18:24:43.798242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.798107 4582 scope.go:122] "RemoveContainer" containerID="b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1" Oct 30 18:24:43.814685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814662 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814687 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814700 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814713 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a90fe9e-b661-483d-a375-d5bfe196be63-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814725 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814736 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814748 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814761 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.814780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814773 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814785 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814797 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814810 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz7km\" (UniqueName: \"kubernetes.io/projected/6a90fe9e-b661-483d-a375-d5bfe196be63-kube-api-access-nz7km\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814838 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4802e94-077b-4c9b-a036-44903c74f610-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814850 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814863 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814876 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814889 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814902 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814915 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814929 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a90fe9e-b661-483d-a375-d5bfe196be63-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814942 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4802e94-077b-4c9b-a036-44903c74f610-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814956 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4802e94-077b-4c9b-a036-44903c74f610-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814970 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814983 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfz6j\" (UniqueName: \"kubernetes.io/projected/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-kube-api-access-lfz6j\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.814996 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.815152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.815009 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4jg7\" (UniqueName: \"kubernetes.io/projected/f4802e94-077b-4c9b-a036-44903c74f610-kube-api-access-c4jg7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.818156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.818131 4582 scope.go:122] "RemoveContainer" containerID="b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1" Oct 30 18:24:43.818419 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:24:43.818401 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1\": container with ID starting with b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1 not found: ID does not exist" containerID="b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1" Oct 30 18:24:43.818496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.818427 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1"} err="failed to get container status \"b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1\": rpc error: code = NotFound desc = could not find container \"b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1\": container with ID starting with b1acdf168aa6150e67e461a423e79cb00db743415804df060561aed8096d37b1 not found: ID does not exist" Oct 30 18:24:43.818496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.818443 4582 scope.go:122] "RemoveContainer" containerID="cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb" Oct 30 18:24:43.835291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.835261 4582 scope.go:122] "RemoveContainer" containerID="cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb" Oct 30 18:24:43.835620 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:24:43.835596 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb\": container with ID starting with cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb not found: ID does not exist" containerID="cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb" Oct 30 18:24:43.835726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.835633 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb"} err="failed to get container status \"cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb\": rpc error: code = NotFound desc = could not find container \"cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb\": container with ID starting with cc3841a77fa6f59f24a09ad41bbc2ad3f2cf5980ff688581162e2a203ff275bb not found: ID does not exist" Oct 30 18:24:43.844221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.844196 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx"] Oct 30 18:24:43.847573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.847552 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4n5fx"] Oct 30 18:24:43.886305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.886273 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p"] Oct 30 18:24:43.889982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.889956 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-v7b8p"] Oct 30 18:24:43.918923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.918893 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl"] Oct 30 18:24:43.921783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:43.921762 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-9rhcl"] Oct 30 18:24:44.576687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:44.576597 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:44.712396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:44.712365 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:44.946275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:44.946238 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.084844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:45.084235 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.300811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:45.300738 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" path="/var/lib/kubelet/pods/6a90fe9e-b661-483d-a375-d5bfe196be63/volumes" Oct 30 18:24:45.301258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:45.301243 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4802e94-077b-4c9b-a036-44903c74f610" path="/var/lib/kubelet/pods/f4802e94-077b-4c9b-a036-44903c74f610/volumes" Oct 30 18:24:45.301662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:45.301650 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" path="/var/lib/kubelet/pods/f8a665a0-c0d5-4c83-bee2-89706cfa8ad8/volumes" Oct 30 18:24:46.440027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.439996 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:46.886465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.886379 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj"] Oct 30 18:24:46.888290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888264 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" Oct 30 18:24:46.888290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888288 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" Oct 30 18:24:46.888491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888313 4582 cpu_manager.go:414] "containerMap: removing container" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" Oct 30 18:24:46.888491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888321 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" Oct 30 18:24:46.888491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888343 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" Oct 30 18:24:46.888491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888351 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" Oct 30 18:24:46.888491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888376 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0e159a0c-4e6b-4243-b880-6ab7f1b77212" containerName="revoke-keys" Oct 30 18:24:46.888491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888383 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e159a0c-4e6b-4243-b880-6ab7f1b77212" containerName="revoke-keys" Oct 30 18:24:46.888810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888525 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4802e94-077b-4c9b-a036-44903c74f610" containerName="istio-proxy" Oct 30 18:24:46.888810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888542 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e159a0c-4e6b-4243-b880-6ab7f1b77212" containerName="revoke-keys" Oct 30 18:24:46.888810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888557 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a665a0-c0d5-4c83-bee2-89706cfa8ad8" containerName="istio-proxy" Oct 30 18:24:46.888810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.888568 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a90fe9e-b661-483d-a375-d5bfe196be63" containerName="istio-proxy" Oct 30 18:24:46.892880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.892856 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.907161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.907142 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-50b278-openshift-default-dockercfg-trcr2\"" Oct 30 18:24:46.930623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.930584 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj"] Oct 30 18:24:46.952544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.952506 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:46.960047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960021 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-envoy\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960057 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5vrc\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-kube-api-access-v5vrc\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960089 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-socket\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960227 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istiod-ca-cert\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960264 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-token\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960391 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-credential-socket\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960440 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-certs\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960498 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-podinfo\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:46.960583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:46.960570 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-data\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.067729 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-credential-socket\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.067813 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-certs\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.067948 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-podinfo\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.068021 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-data\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.068070 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-envoy\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.068104 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5vrc\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-kube-api-access-v5vrc\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.068148 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-socket\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.068230 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istiod-ca-cert\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.068283 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-token\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.069110 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-credential-socket\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.069371 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-certs\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.076841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.070291 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-data\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.085470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.078177 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-socket\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.085470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.078876 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istiod-ca-cert\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.093327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.092216 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-token\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.093327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.092659 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:47.102646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.100337 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-podinfo\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.114707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.113693 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5vrc\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-kube-api-access-v5vrc\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.121879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.115108 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-envoy\") pod \"e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.208157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.208068 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:47.531400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.531330 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj"] Oct 30 18:24:47.534481 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:24:47.534448 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fbc805_f2bd_4dcb_98b7_3d5994e3b9ce.slice/crio-be296c478f58ef719223e87ca59db77d0cf8bfe65b67cf0581a8e6d3cc728b6a WatchSource:0}: Error finding container be296c478f58ef719223e87ca59db77d0cf8bfe65b67cf0581a8e6d3cc728b6a: Status 404 returned error can't find the container with id be296c478f58ef719223e87ca59db77d0cf8bfe65b67cf0581a8e6d3cc728b6a Oct 30 18:24:47.537177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.537138 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:47.537267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.537217 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:47.537267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.537249 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:24:47.557482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.557461 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:47.831545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.831456 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" event={"ID":"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce","Type":"ContainerStarted","Data":"32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f"} Oct 30 18:24:47.831545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.831511 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" event={"ID":"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce","Type":"ContainerStarted","Data":"be296c478f58ef719223e87ca59db77d0cf8bfe65b67cf0581a8e6d3cc728b6a"} Oct 30 18:24:47.871496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:47.871446 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" podStartSLOduration=1.871432354 podStartE2EDuration="1.871432354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:47.867412586 +0000 UTC m=+1584.691806898" watchObservedRunningTime="2026-10-30 18:24:47.871432354 +0000 UTC m=+1584.695826646" Oct 30 18:24:48.208888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:48.208850 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:48.214721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:48.214693 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:48.335057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:48.335015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:48.757528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:48.757493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:48.844918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:48.844833 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:48.846245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:48.846112 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:24:48.884845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:48.876054 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:49.174270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.174241 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:49.646558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.646524 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:49.717292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.717249 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h"] Oct 30 18:24:49.722011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.721983 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.736466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.736438 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:24:49.749402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.746607 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:24:49.752144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.752105 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h"] Oct 30 18:24:49.796309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.796275 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:49.796703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.796382 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:24:49.816259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.816233 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxfg\" (UniqueName: \"kubernetes.io/projected/31cc9516-9569-42b9-9dcf-0244efe4b8d0-kube-api-access-8dxfg\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.816389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.816375 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.816456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.816443 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.918418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.918341 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxfg\" (UniqueName: \"kubernetes.io/projected/31cc9516-9569-42b9-9dcf-0244efe4b8d0-kube-api-access-8dxfg\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.918564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.918440 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.918564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.918479 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.919094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.919075 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.920845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.920792 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:49.935037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:49.935016 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxfg\" (UniqueName: \"kubernetes.io/projected/31cc9516-9569-42b9-9dcf-0244efe4b8d0-kube-api-access-8dxfg\") pod \"maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:50.051016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:50.050985 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:50.322743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:50.322670 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.354814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:50.354782 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h"] Oct 30 18:24:50.358914 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:24:50.358883 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cc9516_9569_42b9_9dcf_0244efe4b8d0.slice/crio-2bc42676e3a917d8ac044d48d097d4b432e6f23f37c00b1b25309fd839a079e6 WatchSource:0}: Error finding container 2bc42676e3a917d8ac044d48d097d4b432e6f23f37c00b1b25309fd839a079e6: Status 404 returned error can't find the container with id 2bc42676e3a917d8ac044d48d097d4b432e6f23f37c00b1b25309fd839a079e6 Oct 30 18:24:50.427146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:50.427105 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.858807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:50.858772 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" event={"ID":"31cc9516-9569-42b9-9dcf-0244efe4b8d0","Type":"ContainerStarted","Data":"ef26a3fb3f7e8a8491e16fcc998600c112343472016f97604a5af40c7ab75b75"} Oct 30 18:24:50.859303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:50.858870 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" event={"ID":"31cc9516-9569-42b9-9dcf-0244efe4b8d0","Type":"ContainerStarted","Data":"2bc42676e3a917d8ac044d48d097d4b432e6f23f37c00b1b25309fd839a079e6"} Oct 30 18:24:50.892896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:50.892849 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" podStartSLOduration=1.892836397 podStartE2EDuration="1.892836397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:50.891871651 +0000 UTC m=+1587.716265943" watchObservedRunningTime="2026-10-30 18:24:50.892836397 +0000 UTC m=+1587.717230679" Oct 30 18:24:51.555572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:51.555491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:51.867575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:51.867487 4582 generic.go:356] "Generic (PLEG): container finished" podID="31cc9516-9569-42b9-9dcf-0244efe4b8d0" containerID="ef26a3fb3f7e8a8491e16fcc998600c112343472016f97604a5af40c7ab75b75" exitCode=0 Oct 30 18:24:51.867575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:51.867549 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" event={"ID":"31cc9516-9569-42b9-9dcf-0244efe4b8d0","Type":"ContainerDied","Data":"ef26a3fb3f7e8a8491e16fcc998600c112343472016f97604a5af40c7ab75b75"} Oct 30 18:24:52.053969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:52.053932 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:52.153615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:52.153584 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:53.039766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.039744 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:53.165999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.165893 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cc9516-9569-42b9-9dcf-0244efe4b8d0-kube-api-access-8dxfg\" (UniqueName: \"kubernetes.io/projected/31cc9516-9569-42b9-9dcf-0244efe4b8d0-kube-api-access-8dxfg\") pod \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " Oct 30 18:24:53.166142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.166067 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-xks-ca\") pod \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " Oct 30 18:24:53.166194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.166187 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-openshift-ca\") pod \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\" (UID: \"31cc9516-9569-42b9-9dcf-0244efe4b8d0\") " Oct 30 18:24:53.166565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.166536 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-openshift-ca" pod "31cc9516-9569-42b9-9dcf-0244efe4b8d0" (UID: "31cc9516-9569-42b9-9dcf-0244efe4b8d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:53.168203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.168170 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cc9516-9569-42b9-9dcf-0244efe4b8d0-kube-api-access-8dxfg" pod "31cc9516-9569-42b9-9dcf-0244efe4b8d0" (UID: "31cc9516-9569-42b9-9dcf-0244efe4b8d0"). InnerVolumeSpecName "kube-api-access-8dxfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:53.168335 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.168297 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-xks-ca" pod "31cc9516-9569-42b9-9dcf-0244efe4b8d0" (UID: "31cc9516-9569-42b9-9dcf-0244efe4b8d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:53.267624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.267591 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:53.267624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.267621 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31cc9516-9569-42b9-9dcf-0244efe4b8d0-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:53.267624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.267632 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dxfg\" (UniqueName: \"kubernetes.io/projected/31cc9516-9569-42b9-9dcf-0244efe4b8d0-kube-api-access-8dxfg\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:53.448698 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.448605 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:53.890035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.889994 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" event={"ID":"31cc9516-9569-42b9-9dcf-0244efe4b8d0","Type":"ContainerDied","Data":"2bc42676e3a917d8ac044d48d097d4b432e6f23f37c00b1b25309fd839a079e6"} Oct 30 18:24:53.890219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.890104 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc42676e3a917d8ac044d48d097d4b432e6f23f37c00b1b25309fd839a079e6" Oct 30 18:24:53.890219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:53.890189 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h" Oct 30 18:24:55.240100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:55.240060 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:56.806154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:56.806122 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:56.901425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:56.901396 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt"] Oct 30 18:24:56.901608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:56.901590 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" podUID="dd25b6ee-bc40-4927-b368-7c7433acb2db" containerName="payload-processing" containerID="cri-o://8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193" gracePeriod=30 Oct 30 18:24:56.921371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:56.921341 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4"] Oct 30 18:24:56.921735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:56.921704 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" podUID="18b9f17f-a075-4f44-88c1-30658d0eaaed" containerName="payload-pre-processing" containerID="cri-o://e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f" gracePeriod=30 Oct 30 18:24:56.991392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:56.991367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.075313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.075186 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.337228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.337156 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:24:57.348071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.348050 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:24:57.382993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.382961 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.452080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.452052 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b9f17f-a075-4f44-88c1-30658d0eaaed-kube-api-access-7fmmf\" (UniqueName: \"kubernetes.io/projected/18b9f17f-a075-4f44-88c1-30658d0eaaed-kube-api-access-7fmmf\") pod \"18b9f17f-a075-4f44-88c1-30658d0eaaed\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " Oct 30 18:24:57.452238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.452180 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd25b6ee-bc40-4927-b368-7c7433acb2db-kube-api-access-bp69c\" (UniqueName: \"kubernetes.io/projected/dd25b6ee-bc40-4927-b368-7c7433acb2db-kube-api-access-bp69c\") pod \"dd25b6ee-bc40-4927-b368-7c7433acb2db\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " Oct 30 18:24:57.452337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.452322 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd25b6ee-bc40-4927-b368-7c7433acb2db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd25b6ee-bc40-4927-b368-7c7433acb2db-plugins-config-volume\") pod \"dd25b6ee-bc40-4927-b368-7c7433acb2db\" (UID: \"dd25b6ee-bc40-4927-b368-7c7433acb2db\") " Oct 30 18:24:57.452386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.452364 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b9f17f-a075-4f44-88c1-30658d0eaaed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b9f17f-a075-4f44-88c1-30658d0eaaed-plugins-config-volume\") pod \"18b9f17f-a075-4f44-88c1-30658d0eaaed\" (UID: \"18b9f17f-a075-4f44-88c1-30658d0eaaed\") " Oct 30 18:24:57.452751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.452724 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd25b6ee-bc40-4927-b368-7c7433acb2db-plugins-config-volume" pod "dd25b6ee-bc40-4927-b368-7c7433acb2db" (UID: "dd25b6ee-bc40-4927-b368-7c7433acb2db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:57.452875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.452795 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b9f17f-a075-4f44-88c1-30658d0eaaed-plugins-config-volume" pod "18b9f17f-a075-4f44-88c1-30658d0eaaed" (UID: "18b9f17f-a075-4f44-88c1-30658d0eaaed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:57.454285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.454255 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b9f17f-a075-4f44-88c1-30658d0eaaed-kube-api-access-7fmmf" pod "18b9f17f-a075-4f44-88c1-30658d0eaaed" (UID: "18b9f17f-a075-4f44-88c1-30658d0eaaed"). InnerVolumeSpecName "kube-api-access-7fmmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:57.454571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.454546 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd25b6ee-bc40-4927-b368-7c7433acb2db-kube-api-access-bp69c" pod "dd25b6ee-bc40-4927-b368-7c7433acb2db" (UID: "dd25b6ee-bc40-4927-b368-7c7433acb2db"). InnerVolumeSpecName "kube-api-access-bp69c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:57.536680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.536646 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.553742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.553712 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmmf\" (UniqueName: \"kubernetes.io/projected/18b9f17f-a075-4f44-88c1-30658d0eaaed-kube-api-access-7fmmf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.553742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.553740 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp69c\" (UniqueName: \"kubernetes.io/projected/dd25b6ee-bc40-4927-b368-7c7433acb2db-kube-api-access-bp69c\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.553916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.553750 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd25b6ee-bc40-4927-b368-7c7433acb2db-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.553916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.553759 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18b9f17f-a075-4f44-88c1-30658d0eaaed-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.588136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.588062 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm"] Oct 30 18:24:57.600145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.600098 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-011d6f-cwscm"] Oct 30 18:24:57.645748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.645718 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.920959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.920927 4582 generic.go:356] "Generic (PLEG): container finished" podID="dd25b6ee-bc40-4927-b368-7c7433acb2db" containerID="8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193" exitCode=0 Oct 30 18:24:57.921372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.920990 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" Oct 30 18:24:57.921372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.921029 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" event={"ID":"dd25b6ee-bc40-4927-b368-7c7433acb2db","Type":"ContainerDied","Data":"8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193"} Oct 30 18:24:57.921372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.921099 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt" event={"ID":"dd25b6ee-bc40-4927-b368-7c7433acb2db","Type":"ContainerDied","Data":"5ab06ddfaefbedb7a0998154fb3d1149d2e3e6c4b15828663f9a6e987f520849"} Oct 30 18:24:57.921372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.921133 4582 scope.go:122] "RemoveContainer" containerID="8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193" Oct 30 18:24:57.923145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.923123 4582 generic.go:356] "Generic (PLEG): container finished" podID="18b9f17f-a075-4f44-88c1-30658d0eaaed" containerID="e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f" exitCode=0 Oct 30 18:24:57.923267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.923167 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" event={"ID":"18b9f17f-a075-4f44-88c1-30658d0eaaed","Type":"ContainerDied","Data":"e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f"} Oct 30 18:24:57.923267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.923192 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" Oct 30 18:24:57.923384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.923198 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4" event={"ID":"18b9f17f-a075-4f44-88c1-30658d0eaaed","Type":"ContainerDied","Data":"ccff782af1bd0fbb6f464f2633bc41db555085db5ac65390f264d2cd28efe503"} Oct 30 18:24:57.935060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.934514 4582 scope.go:122] "RemoveContainer" containerID="8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193" Oct 30 18:24:57.935060 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:24:57.934970 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193\": container with ID starting with 8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193 not found: ID does not exist" containerID="8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193" Oct 30 18:24:57.935060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.935008 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193"} err="failed to get container status \"8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193\": rpc error: code = NotFound desc = could not find container \"8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193\": container with ID starting with 8100c0c00af1ede4958eb8dba01123055dedd44b8f82ebc1cf127c3b7bb00193 not found: ID does not exist" Oct 30 18:24:57.935060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.935030 4582 scope.go:122] "RemoveContainer" containerID="e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f" Oct 30 18:24:57.954142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.954119 4582 scope.go:122] "RemoveContainer" containerID="e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f" Oct 30 18:24:57.954531 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:24:57.954498 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f\": container with ID starting with e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f not found: ID does not exist" containerID="e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f" Oct 30 18:24:57.954586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.954556 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f"} err="failed to get container status \"e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f\": rpc error: code = NotFound desc = could not find container \"e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f\": container with ID starting with e6f53c01526e165a656a04fafcc86464e2728802890e5aae895363810d5afc5f not found: ID does not exist" Oct 30 18:24:57.978079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.978056 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4"] Oct 30 18:24:57.984072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:57.984049 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-011d6f-85479995f5-54sn4"] Oct 30 18:24:58.077176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:58.077142 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:58.077500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:58.077476 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt"] Oct 30 18:24:58.097347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:58.097314 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-011d6f-cc789d486-srcwt"] Oct 30 18:24:59.301787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:59.301756 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e159a0c-4e6b-4243-b880-6ab7f1b77212" path="/var/lib/kubelet/pods/0e159a0c-4e6b-4243-b880-6ab7f1b77212/volumes" Oct 30 18:24:59.302225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:59.302209 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b9f17f-a075-4f44-88c1-30658d0eaaed" path="/var/lib/kubelet/pods/18b9f17f-a075-4f44-88c1-30658d0eaaed/volumes" Oct 30 18:24:59.302581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:59.302568 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd25b6ee-bc40-4927-b368-7c7433acb2db" path="/var/lib/kubelet/pods/dd25b6ee-bc40-4927-b368-7c7433acb2db/volumes" Oct 30 18:24:59.697912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:24:59.697883 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:00.037566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:00.037491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:00.334904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:00.329286 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:00.432341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:00.432302 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:01.941349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:01.941300 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:02.021248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:02.021217 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:02.126589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:02.126556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:04.139201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:04.139153 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:04.280449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:04.280413 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:04.393982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:04.393877 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:05.863463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:05.863428 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:06.225391 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:25:06.225355 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:25:07.558570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:07.558538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:09.947861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:09.947758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:12.061490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:12.061458 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:13.703149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:13.703113 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:13.787581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:13.787547 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:13.902627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:13.902598 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:14.465959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:14.465922 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:14.606033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:14.606000 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:15.196671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.196645 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5"] Oct 30 18:25:15.197255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197239 4582 cpu_manager.go:414] "containerMap: removing container" podUID="dd25b6ee-bc40-4927-b368-7c7433acb2db" containerName="payload-processing" Oct 30 18:25:15.197300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197256 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd25b6ee-bc40-4927-b368-7c7433acb2db" containerName="payload-processing" Oct 30 18:25:15.197300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197283 4582 cpu_manager.go:414] "containerMap: removing container" podUID="18b9f17f-a075-4f44-88c1-30658d0eaaed" containerName="payload-pre-processing" Oct 30 18:25:15.197300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197289 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b9f17f-a075-4f44-88c1-30658d0eaaed" containerName="payload-pre-processing" Oct 30 18:25:15.197300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197298 4582 cpu_manager.go:414] "containerMap: removing container" podUID="31cc9516-9569-42b9-9dcf-0244efe4b8d0" containerName="revoke-keys" Oct 30 18:25:15.197426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197303 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31cc9516-9569-42b9-9dcf-0244efe4b8d0" containerName="revoke-keys" Oct 30 18:25:15.197426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197402 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b9f17f-a075-4f44-88c1-30658d0eaaed" containerName="payload-pre-processing" Oct 30 18:25:15.197426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197409 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="31cc9516-9569-42b9-9dcf-0244efe4b8d0" containerName="revoke-keys" Oct 30 18:25:15.197426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.197423 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd25b6ee-bc40-4927-b368-7c7433acb2db" containerName="payload-processing" Oct 30 18:25:15.201259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.201239 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.217856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.217813 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5"] Oct 30 18:25:15.223499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.223477 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:25:15.223648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.223496 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:15.223648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.223502 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:25:15.223783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.223510 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:25:15.307353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.307321 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.307529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.307394 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.307529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.307505 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g22n\" (UniqueName: \"kubernetes.io/projected/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-kube-api-access-9g22n\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.408551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.408519 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.408726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.408590 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.408726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.408647 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g22n\" (UniqueName: \"kubernetes.io/projected/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-kube-api-access-9g22n\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.409202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.409179 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.411029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.411007 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.425418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.425398 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g22n\" (UniqueName: \"kubernetes.io/projected/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-kube-api-access-9g22n\") pod \"maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.514894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.514808 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:15.701243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.701202 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:15.832898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:15.832863 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5"] Oct 30 18:25:15.835611 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:25:15.835581 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2a4328_3ef8_4007_9a3d_50792e0f38ff.slice/crio-c29deb302368ee5fb397ba979c8b229c4107da7c7602e8f929a0c746b801ee93 WatchSource:0}: Error finding container c29deb302368ee5fb397ba979c8b229c4107da7c7602e8f929a0c746b801ee93: Status 404 returned error can't find the container with id c29deb302368ee5fb397ba979c8b229c4107da7c7602e8f929a0c746b801ee93 Oct 30 18:25:16.059799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:16.059763 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" event={"ID":"fa2a4328-3ef8-4007-9a3d-50792e0f38ff","Type":"ContainerStarted","Data":"c29deb302368ee5fb397ba979c8b229c4107da7c7602e8f929a0c746b801ee93"} Oct 30 18:25:17.071990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:17.071893 4582 generic.go:356] "Generic (PLEG): container finished" podID="fa2a4328-3ef8-4007-9a3d-50792e0f38ff" containerID="e18ccac4106262fff0f538c791267875ce53a258d90767b90104c346edbcbcc3" exitCode=0 Oct 30 18:25:17.072371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:17.071989 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" event={"ID":"fa2a4328-3ef8-4007-9a3d-50792e0f38ff","Type":"ContainerDied","Data":"e18ccac4106262fff0f538c791267875ce53a258d90767b90104c346edbcbcc3"} Oct 30 18:25:17.611874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:17.611810 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:18.248679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.248654 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:18.350891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.350775 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-kube-api-access-9g22n\" (UniqueName: \"kubernetes.io/projected/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-kube-api-access-9g22n\") pod \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " Oct 30 18:25:18.350891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.350889 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-openshift-ca\") pod \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " Oct 30 18:25:18.351095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.351062 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-xks-ca\") pod \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\" (UID: \"fa2a4328-3ef8-4007-9a3d-50792e0f38ff\") " Oct 30 18:25:18.351562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.351536 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-openshift-ca" pod "fa2a4328-3ef8-4007-9a3d-50792e0f38ff" (UID: "fa2a4328-3ef8-4007-9a3d-50792e0f38ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:18.351706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.351688 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:18.353774 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.353751 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-xks-ca" pod "fa2a4328-3ef8-4007-9a3d-50792e0f38ff" (UID: "fa2a4328-3ef8-4007-9a3d-50792e0f38ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:18.353910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.353784 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-kube-api-access-9g22n" pod "fa2a4328-3ef8-4007-9a3d-50792e0f38ff" (UID: "fa2a4328-3ef8-4007-9a3d-50792e0f38ff"). InnerVolumeSpecName "kube-api-access-9g22n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:18.452946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.452909 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:18.452946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:18.452946 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g22n\" (UniqueName: \"kubernetes.io/projected/fa2a4328-3ef8-4007-9a3d-50792e0f38ff-kube-api-access-9g22n\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:19.092355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:19.092328 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" Oct 30 18:25:19.096076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:19.092490 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5" event={"ID":"fa2a4328-3ef8-4007-9a3d-50792e0f38ff","Type":"ContainerDied","Data":"c29deb302368ee5fb397ba979c8b229c4107da7c7602e8f929a0c746b801ee93"} Oct 30 18:25:19.096593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:19.096558 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29deb302368ee5fb397ba979c8b229c4107da7c7602e8f929a0c746b801ee93" Oct 30 18:25:19.426015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:19.425985 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:21.284877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:21.284814 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:22.975134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:22.975095 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:25.166848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:25.164358 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:26.938859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:26.938805 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:28.641267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:28.641230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:29.785973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:29.785899 4582 scope.go:122] "RemoveContainer" containerID="9ad6ca9956f14d65035a3c882f4de30694a40b1eaf4240a8e69647508c2c5a6d" Oct 30 18:25:29.796593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:29.796576 4582 scope.go:122] "RemoveContainer" containerID="e8d9ae517ff30f8db7c3e964ebf6505d4d06d2accd4e3b7dca6692836d0bd379" Oct 30 18:25:29.805881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:29.805864 4582 scope.go:122] "RemoveContainer" containerID="3a66a26ee783c58a1d8d2480cb0be731ba7a9f5bb1dac82eb57a944fad842212" Oct 30 18:25:30.177873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:30.177814 4582 generic.go:356] "Generic (PLEG): container finished" podID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerID="63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2" exitCode=0 Oct 30 18:25:30.178042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:30.177887 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" event={"ID":"94fae2ec-2b00-45dd-b9ef-06decebd68b9","Type":"ContainerDied","Data":"63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2"} Oct 30 18:25:30.178303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:30.178287 4582 scope.go:122] "RemoveContainer" containerID="63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2" Oct 30 18:25:30.180086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:30.180057 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:30.180149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:30.180127 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:30.180185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:30.180162 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:30.416227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:30.416196 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:31.187076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:31.187043 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" event={"ID":"94fae2ec-2b00-45dd-b9ef-06decebd68b9","Type":"ContainerStarted","Data":"8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548"} Oct 30 18:25:31.562569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:31.562478 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:25:31.562917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:31.562892 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:25:31.563087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:31.563061 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 18:25:32.397313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:32.397266 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:32.583414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:32.583383 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:25:32.583703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:32.583646 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 18:25:32.597118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:32.597082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:32.697730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:32.697657 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:33.027912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.026395 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:33.159533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.159479 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:33.261841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.261589 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:33.335666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.335593 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5"] Oct 30 18:25:33.367112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.356997 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-17df15-lcql5"] Oct 30 18:25:33.560842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.559902 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:25:33.560842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.559982 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 18:25:33.736238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:33.736197 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:34.562844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:34.559227 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:25:34.562844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:34.559291 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 18:25:34.696128 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:34.696087 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:25:35.172743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:35.172681 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:35.318159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:35.318124 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2a4328-3ef8-4007-9a3d-50792e0f38ff" path="/var/lib/kubelet/pods/fa2a4328-3ef8-4007-9a3d-50792e0f38ff/volumes" Oct 30 18:25:35.586903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:35.567237 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:25:35.586903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:35.567302 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 18:25:36.229414 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:25:36.229377 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:25:36.656010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:36.655976 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:25:36.668319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:36.668290 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:25:36.968272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:36.968178 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:37.157323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:37.157245 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:38.288289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:38.288259 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8"] Oct 30 18:25:38.288713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:38.288446 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" containerID="cri-o://306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c" gracePeriod=30 Oct 30 18:25:39.063449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:39.063415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:39.672721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:39.672576 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:39.774015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:39.773973 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:39.960854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:39.960730 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn"] Oct 30 18:25:39.961717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:39.961676 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" containerID="cri-o://42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8" gracePeriod=30 Oct 30 18:25:39.974178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:39.973061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:40.852364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:40.852329 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:41.080956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:41.080918 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.257627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:42.257592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.620404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:42.620365 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.746088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:42.746056 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.862160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:42.862124 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.986871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:42.986764 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.347968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.347938 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.647155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.647125 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.656710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.656688 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:25:43.791378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791345 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-certs\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791416 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-token\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-token\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791458 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03abb5af-c9ed-4ba5-b54c-04752276b68f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03abb5af-c9ed-4ba5-b54c-04752276b68f-istiod-ca-cert\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791504 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-data\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791604 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-credential-socket\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791639 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-certs" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.791733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791690 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-podinfo\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791748 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-envoy\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791778 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-kube-api-access-q6f2s\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-kube-api-access-q6f2s\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.791932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791846 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-socket\") pod \"03abb5af-c9ed-4ba5-b54c-04752276b68f\" (UID: \"03abb5af-c9ed-4ba5-b54c-04752276b68f\") " Oct 30 18:25:43.792090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791935 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03abb5af-c9ed-4ba5-b54c-04752276b68f-istiod-ca-cert" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:43.792090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.791971 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-credential-socket" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.792199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.792152 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-data" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.792409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.792391 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-socket" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.792573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.792503 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03abb5af-c9ed-4ba5-b54c-04752276b68f-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.792573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.792520 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.792573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.792533 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.792573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.792547 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.792573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.792561 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.794281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.794258 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-podinfo" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:43.794361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.794279 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-token" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.794568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.794545 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-envoy" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.794628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.794569 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-kube-api-access-q6f2s" pod "03abb5af-c9ed-4ba5-b54c-04752276b68f" (UID: "03abb5af-c9ed-4ba5-b54c-04752276b68f"). InnerVolumeSpecName "kube-api-access-q6f2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.894568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.894534 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.894568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.894569 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.894776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.894583 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03abb5af-c9ed-4ba5-b54c-04752276b68f-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.894776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:43.894595 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6f2s\" (UniqueName: \"kubernetes.io/projected/03abb5af-c9ed-4ba5-b54c-04752276b68f-kube-api-access-q6f2s\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.305805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.305724 4582 generic.go:356] "Generic (PLEG): container finished" podID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerID="32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f" exitCode=0 Oct 30 18:25:44.305805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.305793 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" event={"ID":"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce","Type":"ContainerDied","Data":"32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f"} Oct 30 18:25:44.306276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.306258 4582 scope.go:122] "RemoveContainer" containerID="32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f" Oct 30 18:25:44.307898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.307874 4582 generic.go:356] "Generic (PLEG): container finished" podID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerID="306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c" exitCode=0 Oct 30 18:25:44.308016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.307918 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" Oct 30 18:25:44.308016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.307915 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:44.308016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.307959 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" event={"ID":"03abb5af-c9ed-4ba5-b54c-04752276b68f","Type":"ContainerDied","Data":"306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c"} Oct 30 18:25:44.308016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.307994 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:44.308232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.308021 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8" event={"ID":"03abb5af-c9ed-4ba5-b54c-04752276b68f","Type":"ContainerDied","Data":"60b1f2ce61bbe36540a5f26bf32c27b037dccdba3b911b3822554b8a40537331"} Oct 30 18:25:44.308232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.308030 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:44.308232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.308053 4582 scope.go:122] "RemoveContainer" containerID="306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c" Oct 30 18:25:44.321234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.321215 4582 scope.go:122] "RemoveContainer" containerID="a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1" Oct 30 18:25:44.331793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.331773 4582 scope.go:122] "RemoveContainer" containerID="306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c" Oct 30 18:25:44.332118 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:25:44.332091 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c\": container with ID starting with 306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c not found: ID does not exist" containerID="306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c" Oct 30 18:25:44.332211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.332132 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c"} err="failed to get container status \"306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c\": rpc error: code = NotFound desc = could not find container \"306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c\": container with ID starting with 306a4dac8e4838c4cc01dcfb37d50dbedf9cbd1d4895a253c5bc1102d336162c not found: ID does not exist" Oct 30 18:25:44.332211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.332157 4582 scope.go:122] "RemoveContainer" containerID="a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1" Oct 30 18:25:44.332488 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:25:44.332463 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1\": container with ID starting with a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1 not found: ID does not exist" containerID="a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1" Oct 30 18:25:44.332558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.332494 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1"} err="failed to get container status \"a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1\": rpc error: code = NotFound desc = could not find container \"a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1\": container with ID starting with a48d060f16821dabeae5b7b72e9c81f3ad5be3fbc42ef3649b4102ccfd7706c1 not found: ID does not exist" Oct 30 18:25:44.387223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.387144 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8"] Oct 30 18:25:44.406410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.406373 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-q5rz8"] Oct 30 18:25:44.773668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:44.773633 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:45.274008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.273984 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:25:45.300378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.300347 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" path="/var/lib/kubelet/pods/03abb5af-c9ed-4ba5-b54c-04752276b68f/volumes" Oct 30 18:25:45.316410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.316383 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" event={"ID":"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce","Type":"ContainerStarted","Data":"6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87"} Oct 30 18:25:45.317971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.317949 4582 generic.go:356] "Generic (PLEG): container finished" podID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerID="42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8" exitCode=0 Oct 30 18:25:45.318077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.317989 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" Oct 30 18:25:45.318077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.318021 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" event={"ID":"f10a41fd-276b-4bf5-879d-300ef0977e38","Type":"ContainerDied","Data":"42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8"} Oct 30 18:25:45.318077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.318057 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn" event={"ID":"f10a41fd-276b-4bf5-879d-300ef0977e38","Type":"ContainerDied","Data":"581fbd0bad4cc559ae2b8dbbf538cd21c42210a02be826fe4919cb0956e00173"} Oct 30 18:25:45.318214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.318085 4582 scope.go:122] "RemoveContainer" containerID="42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8" Oct 30 18:25:45.328881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.328863 4582 scope.go:122] "RemoveContainer" containerID="9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08" Oct 30 18:25:45.338651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.338559 4582 scope.go:122] "RemoveContainer" containerID="42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8" Oct 30 18:25:45.338936 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:25:45.338911 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8\": container with ID starting with 42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8 not found: ID does not exist" containerID="42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8" Oct 30 18:25:45.339031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.338947 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8"} err="failed to get container status \"42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8\": rpc error: code = NotFound desc = could not find container \"42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8\": container with ID starting with 42246209833fc8da25a01f9964e06126d3ba3e34d562ecdd43b51f0dc515d6b8 not found: ID does not exist" Oct 30 18:25:45.339031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.338966 4582 scope.go:122] "RemoveContainer" containerID="9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08" Oct 30 18:25:45.339232 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:25:45.339211 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08\": container with ID starting with 9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08 not found: ID does not exist" containerID="9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08" Oct 30 18:25:45.339293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.339237 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08"} err="failed to get container status \"9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08\": rpc error: code = NotFound desc = could not find container \"9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08\": container with ID starting with 9c4e71635fe9dd4361a274b0ada708dde6a4882a444b570499dcb1ddae4fcf08 not found: ID does not exist" Oct 30 18:25:45.417529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417497 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-data\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.417946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417551 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-podinfo\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.417946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417597 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-token\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-token\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.417946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417634 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10a41fd-276b-4bf5-879d-300ef0977e38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a41fd-276b-4bf5-879d-300ef0977e38-istiod-ca-cert\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.417946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417769 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-credential-socket\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.417946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417851 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-data" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:45.417946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417878 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-socket\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.417946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417929 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-kube-api-access-7llvw\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-kube-api-access-7llvw\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.418351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417959 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-envoy\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.418351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.417987 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-certs\") pod \"f10a41fd-276b-4bf5-879d-300ef0977e38\" (UID: \"f10a41fd-276b-4bf5-879d-300ef0977e38\") " Oct 30 18:25:45.418351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418052 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-credential-socket" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:45.418351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418168 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-socket" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:45.418351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418211 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10a41fd-276b-4bf5-879d-300ef0977e38-istiod-ca-cert" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:45.418631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418446 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-certs" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:45.418893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418867 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.419014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418898 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.419014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418913 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.419014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418928 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.419014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.418941 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a41fd-276b-4bf5-879d-300ef0977e38-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.419986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.419963 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-token" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:45.420096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.420028 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-podinfo" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:45.420166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.420150 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-envoy" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:45.420311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.420284 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-kube-api-access-7llvw" pod "f10a41fd-276b-4bf5-879d-300ef0977e38" (UID: "f10a41fd-276b-4bf5-879d-300ef0977e38"). InnerVolumeSpecName "kube-api-access-7llvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:45.520266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.520226 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llvw\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-kube-api-access-7llvw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.520266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.520261 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.520266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.520272 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.520532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.520280 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a41fd-276b-4bf5-879d-300ef0977e38-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:45.673516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.673421 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn"] Oct 30 18:25:45.676601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:45.676575 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-17df15-openshift-default-7cd5d6b88f-nswsn"] Oct 30 18:25:46.208850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:46.208795 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:25:46.210116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:46.210087 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.210277 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:46.210153 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 18:25:47.021170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:47.021133 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:47.140688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:47.140654 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:47.208434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:47.208396 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.208644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:47.208471 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 18:25:47.261667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:47.261632 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:47.311657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:47.311505 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" path="/var/lib/kubelet/pods/f10a41fd-276b-4bf5-879d-300ef0977e38/volumes" Oct 30 18:25:47.437265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:47.437231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:48.209177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:48.208702 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.209177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:48.208775 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 18:25:48.776935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:48.776900 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:48.865850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:48.864916 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:25:48.876163 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:48.876111 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:48.984627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:48.984600 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:49.216397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:49.216361 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:25:49.217131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:49.217101 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 18:25:49.886466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:49.886433 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:49.996183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:49.996107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:50.082727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.082634 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h"] Oct 30 18:25:50.087216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.087169 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:50.091625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.091598 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0ac6bb-s468h"] Oct 30 18:25:50.209166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.209087 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:25:50.209166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.209155 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 18:25:50.531920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.531805 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:50.634436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.634391 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:50.730773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:50.730729 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:51.213798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:51.213770 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:25:51.214690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:51.214669 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:25:51.302048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:51.302017 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31cc9516-9569-42b9-9dcf-0244efe4b8d0" path="/var/lib/kubelet/pods/31cc9516-9569-42b9-9dcf-0244efe4b8d0/volumes" Oct 30 18:25:51.854586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:51.854556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:52.131653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:52.131619 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:53.216159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.216125 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t"] Oct 30 18:25:53.216547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.216335 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" podUID="47123ed0-00ec-47a3-88ad-c88861788157" containerName="discovery" containerID="cri-o://7231aaa1d10f46124942763d36c99563f916925b7aba2364f6ab8ab2ad7b813a" gracePeriod=30 Oct 30 18:25:53.258775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.258749 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s"] Oct 30 18:25:53.259051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.259005 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" containerID="cri-o://8476406987a650e5daea2e43a8a99432078c566f67a6a1e0c361b3185451ad29" gracePeriod=30 Oct 30 18:25:53.261684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.261662 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8"] Oct 30 18:25:53.261906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.261869 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" containerID="cri-o://a976a3b054e1157602345ebbc2d4cbc903a4fed1736305b4101fd7b2c4539837" gracePeriod=30 Oct 30 18:25:53.396125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.396085 4582 generic.go:356] "Generic (PLEG): container finished" podID="47123ed0-00ec-47a3-88ad-c88861788157" containerID="7231aaa1d10f46124942763d36c99563f916925b7aba2364f6ab8ab2ad7b813a" exitCode=0 Oct 30 18:25:53.396297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.396154 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" event={"ID":"47123ed0-00ec-47a3-88ad-c88861788157","Type":"ContainerDied","Data":"7231aaa1d10f46124942763d36c99563f916925b7aba2364f6ab8ab2ad7b813a"} Oct 30 18:25:53.507463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.507439 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:25:53.588588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.588556 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-cacerts\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-cacerts\") pod \"47123ed0-00ec-47a3-88ad-c88861788157\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " Oct 30 18:25:53.588754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.588694 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-ca-configmap\") pod \"47123ed0-00ec-47a3-88ad-c88861788157\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " Oct 30 18:25:53.588754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.588743 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47123ed0-00ec-47a3-88ad-c88861788157-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47123ed0-00ec-47a3-88ad-c88861788157-local-certs\") pod \"47123ed0-00ec-47a3-88ad-c88861788157\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " Oct 30 18:25:53.588873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.588857 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-dns-cert\") pod \"47123ed0-00ec-47a3-88ad-c88861788157\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " Oct 30 18:25:53.588947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.588930 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-istio-token\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-istio-token\") pod \"47123ed0-00ec-47a3-88ad-c88861788157\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " Oct 30 18:25:53.589046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.589029 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-kube-api-access-rwjh5\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-kube-api-access-rwjh5\") pod \"47123ed0-00ec-47a3-88ad-c88861788157\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " Oct 30 18:25:53.589105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.589064 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-kubeconfig\") pod \"47123ed0-00ec-47a3-88ad-c88861788157\" (UID: \"47123ed0-00ec-47a3-88ad-c88861788157\") " Oct 30 18:25:53.589165 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.589121 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-ca-configmap" pod "47123ed0-00ec-47a3-88ad-c88861788157" (UID: "47123ed0-00ec-47a3-88ad-c88861788157"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:53.589678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.589645 4582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-ca-configmap\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.592029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.591931 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47123ed0-00ec-47a3-88ad-c88861788157-local-certs" pod "47123ed0-00ec-47a3-88ad-c88861788157" (UID: "47123ed0-00ec-47a3-88ad-c88861788157"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:53.592029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.592020 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-cacerts" pod "47123ed0-00ec-47a3-88ad-c88861788157" (UID: "47123ed0-00ec-47a3-88ad-c88861788157"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:53.592174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.592041 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-istio-token" pod "47123ed0-00ec-47a3-88ad-c88861788157" (UID: "47123ed0-00ec-47a3-88ad-c88861788157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:53.592174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.592073 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-kube-api-access-rwjh5" pod "47123ed0-00ec-47a3-88ad-c88861788157" (UID: "47123ed0-00ec-47a3-88ad-c88861788157"). InnerVolumeSpecName "kube-api-access-rwjh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:53.592933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.592909 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-dns-cert" pod "47123ed0-00ec-47a3-88ad-c88861788157" (UID: "47123ed0-00ec-47a3-88ad-c88861788157"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:53.593012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.592934 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-kubeconfig" pod "47123ed0-00ec-47a3-88ad-c88861788157" (UID: "47123ed0-00ec-47a3-88ad-c88861788157"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:53.691071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.691039 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.691071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.691068 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwjh5\" (UniqueName: \"kubernetes.io/projected/47123ed0-00ec-47a3-88ad-c88861788157-kube-api-access-rwjh5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.691071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.691079 4582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-kubeconfig\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.691318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.691088 4582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-cacerts\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.691318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.691097 4582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47123ed0-00ec-47a3-88ad-c88861788157-local-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.691318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.691106 4582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47123ed0-00ec-47a3-88ad-c88861788157-istio-csr-dns-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.697994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.697968 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:53.977384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:53.977348 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h_15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72/payload-processing/0.log" Oct 30 18:25:54.191987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:54.191957 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log" Oct 30 18:25:54.404206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:54.404179 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" Oct 30 18:25:54.404644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:54.404226 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t" event={"ID":"47123ed0-00ec-47a3-88ad-c88861788157","Type":"ContainerDied","Data":"bf255da526a98f3be7da135bf78f312d6635ce47180179301d567a133c1b6295"} Oct 30 18:25:54.404644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:54.404283 4582 scope.go:122] "RemoveContainer" containerID="7231aaa1d10f46124942763d36c99563f916925b7aba2364f6ab8ab2ad7b813a" Oct 30 18:25:54.414295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:54.414276 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log" Oct 30 18:25:54.457285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:54.457261 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t"] Oct 30 18:25:54.465977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:54.465958 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4k2t"] Oct 30 18:25:55.300336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.300302 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47123ed0-00ec-47a3-88ad-c88861788157" path="/var/lib/kubelet/pods/47123ed0-00ec-47a3-88ad-c88861788157/volumes" Oct 30 18:25:55.577364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577288 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj"] Oct 30 18:25:55.577865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577850 4582 cpu_manager.go:414] "containerMap: removing container" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" Oct 30 18:25:55.577865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577865 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577880 4582 cpu_manager.go:414] "containerMap: removing container" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577885 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577899 4582 cpu_manager.go:414] "containerMap: removing container" podUID="47123ed0-00ec-47a3-88ad-c88861788157" containerName="discovery" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577905 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47123ed0-00ec-47a3-88ad-c88861788157" containerName="discovery" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577917 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577921 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577942 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" Oct 30 18:25:55.577953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577947 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" Oct 30 18:25:55.578190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577957 4582 cpu_manager.go:414] "containerMap: removing container" podUID="fa2a4328-3ef8-4007-9a3d-50792e0f38ff" containerName="revoke-keys" Oct 30 18:25:55.578190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.577962 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2a4328-3ef8-4007-9a3d-50792e0f38ff" containerName="revoke-keys" Oct 30 18:25:55.578190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.578056 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="47123ed0-00ec-47a3-88ad-c88861788157" containerName="discovery" Oct 30 18:25:55.578190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.578066 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" Oct 30 18:25:55.578190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.578076 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2a4328-3ef8-4007-9a3d-50792e0f38ff" containerName="revoke-keys" Oct 30 18:25:55.578190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.578085 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" Oct 30 18:25:55.578190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.578093 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="03abb5af-c9ed-4ba5-b54c-04752276b68f" containerName="istio-proxy" Oct 30 18:25:55.581444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.581428 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.598784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.598757 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj"] Oct 30 18:25:55.604893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.604868 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:55.605026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.604904 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:25:55.605026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.604952 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:25:55.605026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.604987 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:25:55.719062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.719024 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.719242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.719175 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqk7\" (UniqueName: \"kubernetes.io/projected/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-kube-api-access-rpqk7\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.719327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.719302 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.820553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.820517 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.820744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.820571 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.820744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.820663 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqk7\" (UniqueName: \"kubernetes.io/projected/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-kube-api-access-rpqk7\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.821178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.821153 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.822993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.822968 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.837541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.837481 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqk7\" (UniqueName: \"kubernetes.io/projected/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-kube-api-access-rpqk7\") pod \"maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:55.894087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:55.894061 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:56.071020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.070984 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:56.190212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.190107 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj"] Oct 30 18:25:56.193285 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:25:56.193252 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d14bb58_c0be_413e_b24b_c9cc3d8606d3.slice/crio-154730bb61c571499355ed076cf6699dd7a24be1c51108c491dd8979d6ec83c3 WatchSource:0}: Error finding container 154730bb61c571499355ed076cf6699dd7a24be1c51108c491dd8979d6ec83c3: Status 404 returned error can't find the container with id 154730bb61c571499355ed076cf6699dd7a24be1c51108c491dd8979d6ec83c3 Oct 30 18:25:56.429256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.429223 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" event={"ID":"6d14bb58-c0be-413e-b24b-c9cc3d8606d3","Type":"ContainerStarted","Data":"154730bb61c571499355ed076cf6699dd7a24be1c51108c491dd8979d6ec83c3"} Oct 30 18:25:56.554409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.554376 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:56.659098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.659063 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:56.926796 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.926761 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl"] Oct 30 18:25:56.927066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.927040 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" containerID="cri-o://891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89" gracePeriod=30 Oct 30 18:25:56.965248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:56.965214 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:57.245927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:57.245838 4582 patch_prober.go:29] interesting pod/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:57.245927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:57.245903 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:57.438073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:57.438039 4582 generic.go:356] "Generic (PLEG): container finished" podID="6d14bb58-c0be-413e-b24b-c9cc3d8606d3" containerID="96ea05b2652f871411890bdd46e447bf900707bf846dc5592c0ac60a16036c53" exitCode=0 Oct 30 18:25:57.438241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:57.438112 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" event={"ID":"6d14bb58-c0be-413e-b24b-c9cc3d8606d3","Type":"ContainerDied","Data":"96ea05b2652f871411890bdd46e447bf900707bf846dc5592c0ac60a16036c53"} Oct 30 18:25:57.854648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:57.854618 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.218242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.218208 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.285961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.285932 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5"] Oct 30 18:25:58.287078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.287055 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10a41fd-276b-4bf5-879d-300ef0977e38" containerName="istio-proxy" Oct 30 18:25:58.297151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.297128 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.308193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.308169 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5"] Oct 30 18:25:58.314616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.314581 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e9b146-openshift-default-dockercfg-9ps2d\"" Oct 30 18:25:58.322140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.322031 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.421744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.421715 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.459874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.459645 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-socket\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.459874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.459702 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-token\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.459874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.459791 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzh4l\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-kube-api-access-pzh4l\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.522999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.494374 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-podinfo\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.522999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.494512 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-credential-socket\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.522999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.494569 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-data\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.522999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.494613 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-envoy\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.522999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.494695 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-certs\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.522999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.494741 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50651ef2-6ef6-41d3-ab69-893124fb312b-istiod-ca-cert\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.556075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.554604 4582 generic.go:356] "Generic (PLEG): container finished" podID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerID="8476406987a650e5daea2e43a8a99432078c566f67a6a1e0c361b3185451ad29" exitCode=0 Oct 30 18:25:58.556075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.554635 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" event={"ID":"935d946a-ae03-4628-970a-4e0ca04d89d7","Type":"ContainerDied","Data":"8476406987a650e5daea2e43a8a99432078c566f67a6a1e0c361b3185451ad29"} Oct 30 18:25:58.557744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.557716 4582 generic.go:356] "Generic (PLEG): container finished" podID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerID="a976a3b054e1157602345ebbc2d4cbc903a4fed1736305b4101fd7b2c4539837" exitCode=0 Oct 30 18:25:58.558061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.557787 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" event={"ID":"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6","Type":"ContainerDied","Data":"a976a3b054e1157602345ebbc2d4cbc903a4fed1736305b4101fd7b2c4539837"} Oct 30 18:25:58.596506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596464 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-socket\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596528 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-token\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596600 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzh4l\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-kube-api-access-pzh4l\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596638 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-podinfo\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596710 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-credential-socket\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596757 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-data\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596797 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-envoy\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596877 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-certs\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.596919 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50651ef2-6ef6-41d3-ab69-893124fb312b-istiod-ca-cert\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.597777 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50651ef2-6ef6-41d3-ab69-893124fb312b-istiod-ca-cert\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.599135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.598125 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-socket\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.601042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.600291 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-certs\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.601042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.600433 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-credential-socket\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.601042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.600658 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-data\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.603059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.603032 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-envoy\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.605273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.605240 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-podinfo\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.616172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.615977 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-token\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.619362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.619312 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzh4l\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-kube-api-access-pzh4l\") pod \"e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.650633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.650603 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:25:58.656338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.656318 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:25:58.699493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.699459 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:25:58.710495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.710470 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:58.799918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.799518 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-kube-api-access-rpqk7\" (UniqueName: \"kubernetes.io/projected/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-kube-api-access-rpqk7\") pod \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " Oct 30 18:25:58.799918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.799618 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-data\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.799918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.799883 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-certs" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.800235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.800214 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-certs\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.800285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.800275 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-data\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.800329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.800315 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-kube-api-access-86sd8\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-kube-api-access-86sd8\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.800370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.800346 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-token\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-token\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.800447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.800421 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-podinfo\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.800498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.800463 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-socket\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.802166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.801120 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-data" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.802166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.801635 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-socket" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.802166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.801877 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-data" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.803730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.803571 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-envoy\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.803730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.803612 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-credential-socket\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.804119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804094 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-token" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:58.804202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804172 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-kube-api-access-rpqk7" pod "6d14bb58-c0be-413e-b24b-c9cc3d8606d3" (UID: "6d14bb58-c0be-413e-b24b-c9cc3d8606d3"). InnerVolumeSpecName "kube-api-access-rpqk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:58.804268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804234 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-credential-socket" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.804592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804568 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-openshift-ca" pod "6d14bb58-c0be-413e-b24b-c9cc3d8606d3" (UID: "6d14bb58-c0be-413e-b24b-c9cc3d8606d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:58.804669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804616 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-openshift-ca\") pod \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " Oct 30 18:25:58.804728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804674 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-podinfo\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.804728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804716 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-certs\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.804973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804759 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-xks-ca\") pod \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\" (UID: \"6d14bb58-c0be-413e-b24b-c9cc3d8606d3\") " Oct 30 18:25:58.804973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804793 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-kube-api-access-q995b\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-kube-api-access-q995b\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.804973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804833 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-token\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-token\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.804973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804866 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istiod-ca-cert\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.804973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804924 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935d946a-ae03-4628-970a-4e0ca04d89d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935d946a-ae03-4628-970a-4e0ca04d89d7-istiod-ca-cert\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.804973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804953 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-envoy\") pod \"935d946a-ae03-4628-970a-4e0ca04d89d7\" (UID: \"935d946a-ae03-4628-970a-4e0ca04d89d7\") " Oct 30 18:25:58.805326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.804999 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-socket\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.805326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.805046 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-credential-socket\") pod \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\" (UID: \"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6\") " Oct 30 18:25:58.805514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.805490 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-envoy" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.805862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.805837 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935d946a-ae03-4628-970a-4e0ca04d89d7-istiod-ca-cert" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:58.805999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.805908 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-kube-api-access-86sd8" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "kube-api-access-86sd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:58.806111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.805991 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-podinfo" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:58.806186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806148 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-socket" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806391 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806419 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806433 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806445 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86sd8\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-kube-api-access-86sd8\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806459 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806472 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806484 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806496 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806510 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806526 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806540 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935d946a-ae03-4628-970a-4e0ca04d89d7-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806553 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpqk7\" (UniqueName: \"kubernetes.io/projected/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-kube-api-access-rpqk7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806200 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-credential-socket" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806208 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istiod-ca-cert" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:58.806601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.806219 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-certs" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.809227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.809202 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-token" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:58.810369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.810333 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-xks-ca" pod "6d14bb58-c0be-413e-b24b-c9cc3d8606d3" (UID: "6d14bb58-c0be-413e-b24b-c9cc3d8606d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:58.810471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.810409 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-kube-api-access-q995b" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "kube-api-access-q995b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:58.810471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.810464 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-envoy" pod "935d946a-ae03-4628-970a-4e0ca04d89d7" (UID: "935d946a-ae03-4628-970a-4e0ca04d89d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:58.811575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.811556 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-podinfo" pod "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" (UID: "cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:58.907526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907499 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907524 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935d946a-ae03-4628-970a-4e0ca04d89d7-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907533 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907542 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907552 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907562 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907570 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d14bb58-c0be-413e-b24b-c9cc3d8606d3-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907578 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q995b\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-kube-api-access-q995b\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.907974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:58.907585 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.027242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.027189 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5"] Oct 30 18:25:59.030277 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:25:59.030245 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50651ef2_6ef6_41d3_ab69_893124fb312b.slice/crio-9d697d6cedb92e29bbea734d0309886072e4472d8937fde61535422c53308ba0 WatchSource:0}: Error finding container 9d697d6cedb92e29bbea734d0309886072e4472d8937fde61535422c53308ba0: Status 404 returned error can't find the container with id 9d697d6cedb92e29bbea734d0309886072e4472d8937fde61535422c53308ba0 Oct 30 18:25:59.032914 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.032881 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:59.033031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.032948 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:59.033031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.032978 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:25:59.566297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.566211 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" Oct 30 18:25:59.566441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.566218 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s" event={"ID":"935d946a-ae03-4628-970a-4e0ca04d89d7","Type":"ContainerDied","Data":"255d4cd0e7a4e1d4431bc5c068a2b288f546206ecbe4f89e5eabfc65f6dbbd53"} Oct 30 18:25:59.566441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.566370 4582 scope.go:122] "RemoveContainer" containerID="8476406987a650e5daea2e43a8a99432078c566f67a6a1e0c361b3185451ad29" Oct 30 18:25:59.568518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.568491 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" Oct 30 18:25:59.569119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.569097 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8" event={"ID":"cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6","Type":"ContainerDied","Data":"086a02701f74d86d186e71a9f37af9ec2f5f614d747383fa34920cb63a41b94d"} Oct 30 18:25:59.570312 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.570268 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" Oct 30 18:25:59.570312 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.570276 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj" event={"ID":"6d14bb58-c0be-413e-b24b-c9cc3d8606d3","Type":"ContainerDied","Data":"154730bb61c571499355ed076cf6699dd7a24be1c51108c491dd8979d6ec83c3"} Oct 30 18:25:59.570661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.570319 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154730bb61c571499355ed076cf6699dd7a24be1c51108c491dd8979d6ec83c3" Oct 30 18:25:59.572115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.572090 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" event={"ID":"50651ef2-6ef6-41d3-ab69-893124fb312b","Type":"ContainerStarted","Data":"8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893"} Oct 30 18:25:59.572600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.572130 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" event={"ID":"50651ef2-6ef6-41d3-ab69-893124fb312b","Type":"ContainerStarted","Data":"9d697d6cedb92e29bbea734d0309886072e4472d8937fde61535422c53308ba0"} Oct 30 18:25:59.578515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.578498 4582 scope.go:122] "RemoveContainer" containerID="a976a3b054e1157602345ebbc2d4cbc903a4fed1736305b4101fd7b2c4539837" Oct 30 18:25:59.610904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.610881 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8"] Oct 30 18:25:59.619731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.619709 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-89vg8"] Oct 30 18:25:59.655301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.649365 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podStartSLOduration=1.6493486389999998 podStartE2EDuration="1.649348639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:59.645039571 +0000 UTC m=+1656.469433864" watchObservedRunningTime="2026-10-30 18:25:59.649348639 +0000 UTC m=+1656.473742936" Oct 30 18:25:59.678513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.678482 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s"] Oct 30 18:25:59.682226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.682150 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-ljm4s"] Oct 30 18:25:59.699782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:25:59.699760 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:26:00.070898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:00.070859 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:00.705268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:00.705235 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:26:01.300459 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:01.300425 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" path="/var/lib/kubelet/pods/935d946a-ae03-4628-970a-4e0ca04d89d7/volumes" Oct 30 18:26:01.300941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:01.300927 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" path="/var/lib/kubelet/pods/cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6/volumes" Oct 30 18:26:01.587794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:01.587722 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:26:01.588743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:01.588726 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:26:01.917037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:01.916999 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:02.038535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.038503 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:02.185176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.185117 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:26:02.228867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.228839 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:02.335804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.335759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:02.350933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.350908 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-certs\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.350952 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-token\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.350993 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-podinfo\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351015 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-socket\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351147 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-envoy\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351182 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-kube-api-access-2wsns\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-kube-api-access-2wsns\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351284 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351199 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-certs" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:02.351284 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351227 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istiod-ca-cert\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351284 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351261 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-data\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351287 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-credential-socket\") pod \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\" (UID: \"a0cc9b03-2f67-445f-9e2d-be1a776ee428\") " Oct 30 18:26:02.351816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351692 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-data" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:02.351816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351738 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-credential-socket" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:02.351816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351781 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istiod-ca-cert" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:02.351816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.351788 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-socket" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:02.352131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.352047 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.352131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.352070 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.352131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.352084 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.352131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.352102 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.352131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.352115 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.353509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.353485 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-kube-api-access-2wsns" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "kube-api-access-2wsns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:02.353601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.353574 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-podinfo" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:02.353777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.353749 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-envoy" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:02.353900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.353779 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-token" pod "a0cc9b03-2f67-445f-9e2d-be1a776ee428" (UID: "a0cc9b03-2f67-445f-9e2d-be1a776ee428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:02.453664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.453593 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.453664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.453617 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wsns\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-kube-api-access-2wsns\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.453664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.453628 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.453664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.453637 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc9b03-2f67-445f-9e2d-be1a776ee428-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.598125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.598080 4582 generic.go:356] "Generic (PLEG): container finished" podID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerID="891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89" exitCode=0 Oct 30 18:26:02.598315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.598136 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" Oct 30 18:26:02.598315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.598154 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" event={"ID":"a0cc9b03-2f67-445f-9e2d-be1a776ee428","Type":"ContainerDied","Data":"891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89"} Oct 30 18:26:02.598315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.598211 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl" event={"ID":"a0cc9b03-2f67-445f-9e2d-be1a776ee428","Type":"ContainerDied","Data":"8f00284e75e133e9c9256b988b01ef15abe9e41ad0f665cc41647b20edd2f067"} Oct 30 18:26:02.598315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.598239 4582 scope.go:122] "RemoveContainer" containerID="891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89" Oct 30 18:26:02.609588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.609569 4582 scope.go:122] "RemoveContainer" containerID="83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e" Oct 30 18:26:02.618918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.618900 4582 scope.go:122] "RemoveContainer" containerID="891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89" Oct 30 18:26:02.619175 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:02.619157 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89\": container with ID starting with 891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89 not found: ID does not exist" containerID="891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89" Oct 30 18:26:02.619251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.619185 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89"} err="failed to get container status \"891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89\": rpc error: code = NotFound desc = could not find container \"891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89\": container with ID starting with 891c5a722a7c44a2f99a0b7ef25a45361010ed950f7857384e0d889942477c89 not found: ID does not exist" Oct 30 18:26:02.619251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.619202 4582 scope.go:122] "RemoveContainer" containerID="83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e" Oct 30 18:26:02.619436 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:02.619416 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e\": container with ID starting with 83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e not found: ID does not exist" containerID="83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e" Oct 30 18:26:02.619497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.619443 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e"} err="failed to get container status \"83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e\": rpc error: code = NotFound desc = could not find container \"83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e\": container with ID starting with 83eb79a1dff44384aee9e4daf9e3d1f5e2b6b1fd7a8063da59d017a98898280e not found: ID does not exist" Oct 30 18:26:02.649187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.649159 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl"] Oct 30 18:26:02.663302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:02.663269 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0ac6bb-openshift-default-745b898b59-7cbkl"] Oct 30 18:26:03.301309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.301277 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" path="/var/lib/kubelet/pods/a0cc9b03-2f67-445f-9e2d-be1a776ee428/volumes" Oct 30 18:26:03.784910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.784876 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:03.882187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.882155 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:03.901452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.901424 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj"] Oct 30 18:26:03.902016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902002 4582 cpu_manager.go:414] "containerMap: removing container" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" Oct 30 18:26:03.902061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902019 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" Oct 30 18:26:03.902061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902034 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" Oct 30 18:26:03.902061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902039 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" Oct 30 18:26:03.902061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902052 4582 cpu_manager.go:414] "containerMap: removing container" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" Oct 30 18:26:03.902061 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902058 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" Oct 30 18:26:03.902231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902099 4582 cpu_manager.go:414] "containerMap: removing container" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" Oct 30 18:26:03.902231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902107 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" Oct 30 18:26:03.902231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902125 4582 cpu_manager.go:414] "containerMap: removing container" podUID="6d14bb58-c0be-413e-b24b-c9cc3d8606d3" containerName="revoke-keys" Oct 30 18:26:03.902231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902130 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d14bb58-c0be-413e-b24b-c9cc3d8606d3" containerName="revoke-keys" Oct 30 18:26:03.902231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902214 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" Oct 30 18:26:03.902231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902224 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d14bb58-c0be-413e-b24b-c9cc3d8606d3" containerName="revoke-keys" Oct 30 18:26:03.902231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902232 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cead82e5-79e5-4eac-a6f3-e4b09f3c2fb6" containerName="istio-proxy" Oct 30 18:26:03.902444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.902241 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="935d946a-ae03-4628-970a-4e0ca04d89d7" containerName="istio-proxy" Oct 30 18:26:03.905466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.905447 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:03.917918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.917901 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e9b146\"" Oct 30 18:26:03.927258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.927240 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e9b146-dockercfg-7pwwr\"" Oct 30 18:26:03.930709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.930685 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj"] Oct 30 18:26:03.971120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.971089 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx"] Oct 30 18:26:03.971750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.971736 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cc9b03-2f67-445f-9e2d-be1a776ee428" containerName="istio-proxy" Oct 30 18:26:03.974032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:03.974013 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.006017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.005976 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx"] Oct 30 18:26:04.074357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.074271 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38246359-a129-4c55-bccc-3979b7baf132-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.074535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.074414 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtg2\" (UniqueName: \"kubernetes.io/projected/89cacc01-2b2b-496c-aa16-4a5f92fbca16-kube-api-access-9rtg2\") pod \"payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:04.074535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.074513 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86f9\" (UniqueName: \"kubernetes.io/projected/38246359-a129-4c55-bccc-3979b7baf132-kube-api-access-z86f9\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.074664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.074636 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cacc01-2b2b-496c-aa16-4a5f92fbca16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:04.176020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.175987 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtg2\" (UniqueName: \"kubernetes.io/projected/89cacc01-2b2b-496c-aa16-4a5f92fbca16-kube-api-access-9rtg2\") pod \"payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:04.176175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.176075 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86f9\" (UniqueName: \"kubernetes.io/projected/38246359-a129-4c55-bccc-3979b7baf132-kube-api-access-z86f9\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.176175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.176113 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cacc01-2b2b-496c-aa16-4a5f92fbca16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:04.176312 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.176189 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38246359-a129-4c55-bccc-3979b7baf132-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.176886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.176867 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38246359-a129-4c55-bccc-3979b7baf132-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.176923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.176878 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cacc01-2b2b-496c-aa16-4a5f92fbca16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:04.195304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.195276 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtg2\" (UniqueName: \"kubernetes.io/projected/89cacc01-2b2b-496c-aa16-4a5f92fbca16-kube-api-access-9rtg2\") pod \"payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:04.195427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.195279 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86f9\" (UniqueName: \"kubernetes.io/projected/38246359-a129-4c55-bccc-3979b7baf132-kube-api-access-z86f9\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.218009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.217975 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:04.286496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.286464 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:04.451001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.450967 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:04.525093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.525068 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj"] Oct 30 18:26:04.527199 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:26:04.527171 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cacc01_2b2b_496c_aa16_4a5f92fbca16.slice/crio-07d19210fdb0a930e974d91a9d4360503304369c4e8b2e1315920d2860b15ac4 WatchSource:0}: Error finding container 07d19210fdb0a930e974d91a9d4360503304369c4e8b2e1315920d2860b15ac4: Status 404 returned error can't find the container with id 07d19210fdb0a930e974d91a9d4360503304369c4e8b2e1315920d2860b15ac4 Oct 30 18:26:04.588987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.588941 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx"] Oct 30 18:26:04.591845 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:26:04.591762 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38246359_a129_4c55_bccc_3979b7baf132.slice/crio-efe34249c91665f5f1516aa4df8d2621cfffd87a23f3b964c3d891c446591621 WatchSource:0}: Error finding container efe34249c91665f5f1516aa4df8d2621cfffd87a23f3b964c3d891c446591621: Status 404 returned error can't find the container with id efe34249c91665f5f1516aa4df8d2621cfffd87a23f3b964c3d891c446591621 Oct 30 18:26:04.615279 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.615248 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" event={"ID":"38246359-a129-4c55-bccc-3979b7baf132","Type":"ContainerStarted","Data":"efe34249c91665f5f1516aa4df8d2621cfffd87a23f3b964c3d891c446591621"} Oct 30 18:26:04.616812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.616778 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" event={"ID":"89cacc01-2b2b-496c-aa16-4a5f92fbca16","Type":"ContainerStarted","Data":"e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a"} Oct 30 18:26:04.616922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.616846 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" event={"ID":"89cacc01-2b2b-496c-aa16-4a5f92fbca16","Type":"ContainerStarted","Data":"07d19210fdb0a930e974d91a9d4360503304369c4e8b2e1315920d2860b15ac4"} Oct 30 18:26:04.653607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:04.653559 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" podStartSLOduration=1.653544734 podStartE2EDuration="1.653544734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:04.651177545 +0000 UTC m=+1661.475571837" watchObservedRunningTime="2026-10-30 18:26:04.653544734 +0000 UTC m=+1661.477939026" Oct 30 18:26:05.625387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:05.625353 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" event={"ID":"38246359-a129-4c55-bccc-3979b7baf132","Type":"ContainerStarted","Data":"19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88"} Oct 30 18:26:05.625841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:05.625451 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:05.625841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:05.625579 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:05.660554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:05.660511 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" podStartSLOduration=2.66049864 podStartE2EDuration="2.66049864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:05.656392388 +0000 UTC m=+1662.480786681" watchObservedRunningTime="2026-10-30 18:26:05.66049864 +0000 UTC m=+1662.484892931" Oct 30 18:26:06.225869 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:06.225797 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:26:06.442489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:06.442454 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:08.346657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:08.346627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.624294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:10.624264 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.704328 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:10.704293 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.908498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:10.908424 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:12.519607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.519577 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm"] Oct 30 18:26:12.523937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.523910 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.537775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.537753 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-be7f9e-serving-cert\"" Oct 30 18:26:12.547432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.547404 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wjb5t\"" Oct 30 18:26:12.547572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.547442 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:26:12.547669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.547650 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:26:12.552772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.552749 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm"] Oct 30 18:26:12.552910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.552859 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:26:12.578653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.578625 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-metrics-certs\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.578804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.578684 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.579025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.578999 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09911906-7771-4ad2-b09d-6d0d3f6f5008-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.579099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.579063 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hfw\" (UniqueName: \"kubernetes.io/projected/09911906-7771-4ad2-b09d-6d0d3f6f5008-kube-api-access-g7hfw\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.606557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.606512 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:12.680879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.680815 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09911906-7771-4ad2-b09d-6d0d3f6f5008-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.681052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.680895 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hfw\" (UniqueName: \"kubernetes.io/projected/09911906-7771-4ad2-b09d-6d0d3f6f5008-kube-api-access-g7hfw\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.681052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.681004 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-metrics-certs\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.681126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.681055 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.681501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.681475 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09911906-7771-4ad2-b09d-6d0d3f6f5008-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.683492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.683468 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-metrics-certs\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.683612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.683507 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.693372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.693344 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:12.697370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.697345 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hfw\" (UniqueName: \"kubernetes.io/projected/09911906-7771-4ad2-b09d-6d0d3f6f5008-kube-api-access-g7hfw\") pod \"maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:12.839041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:12.838967 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:13.145269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:13.145228 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm"] Oct 30 18:26:13.152926 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:26:13.152895 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09911906_7771_4ad2_b09d_6d0d3f6f5008.slice/crio-9062269a798c83da9dc931a08929719c929d512707eec182e5823bfec20d9109 WatchSource:0}: Error finding container 9062269a798c83da9dc931a08929719c929d512707eec182e5823bfec20d9109: Status 404 returned error can't find the container with id 9062269a798c83da9dc931a08929719c929d512707eec182e5823bfec20d9109 Oct 30 18:26:13.154338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:13.154313 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:26:13.681864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:13.681814 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" event={"ID":"09911906-7771-4ad2-b09d-6d0d3f6f5008","Type":"ContainerStarted","Data":"9062269a798c83da9dc931a08929719c929d512707eec182e5823bfec20d9109"} Oct 30 18:26:13.788601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:13.788560 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:13.882146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:13.882112 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:14.462305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:14.462258 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:14.583637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:14.583603 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:14.691189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:14.691158 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" event={"ID":"09911906-7771-4ad2-b09d-6d0d3f6f5008","Type":"ContainerStarted","Data":"5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a"} Oct 30 18:26:14.691656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:14.691356 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:14.732815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:14.732724 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" podStartSLOduration=1.282999424 podStartE2EDuration="2.732705669s" totalImagesPullingTime="1.449706245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:14.731174971 +0000 UTC m=+1671.555569275" watchObservedRunningTime="2026-10-30 18:26:14.732705669 +0000 UTC m=+1671.557099960" Oct 30 18:26:16.199021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:16.198991 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:16.635430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:16.635407 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:26:16.635586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:16.635556 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:26:17.868290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:17.868256 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:19.318976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.318942 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg"] Oct 30 18:26:19.323124 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.323105 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.338706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.338684 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:19.341478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.341456 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg"] Oct 30 18:26:19.347598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.347574 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:26:19.371438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.371403 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8pc\" (UniqueName: \"kubernetes.io/projected/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-kube-api-access-sx8pc\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.371627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.371457 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.371627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.371568 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.473147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.473115 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.473315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.473186 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.473383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.473332 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8pc\" (UniqueName: \"kubernetes.io/projected/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-kube-api-access-sx8pc\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.473899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.473877 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.475434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.475409 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.490714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.490686 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8pc\" (UniqueName: \"kubernetes.io/projected/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-kube-api-access-sx8pc\") pod \"maas-api-revoke-sub-847168d2-xbmlg\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.636917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.636888 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:19.658546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.658513 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:19.938985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:19.938914 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg"] Oct 30 18:26:19.942687 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:26:19.942652 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f3e58a_33af_4e84_8dea_7a4c9f9ae2bd.slice/crio-7cce503b736d3e5352f8e901ef69d8fc7acb3e066bc2f3fb9a9df0d9e2529f98 WatchSource:0}: Error finding container 7cce503b736d3e5352f8e901ef69d8fc7acb3e066bc2f3fb9a9df0d9e2529f98: Status 404 returned error can't find the container with id 7cce503b736d3e5352f8e901ef69d8fc7acb3e066bc2f3fb9a9df0d9e2529f98 Oct 30 18:26:20.703461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:20.703434 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:26:20.766469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:20.766435 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" event={"ID":"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd","Type":"ContainerStarted","Data":"cd7132eccf2952d0db631f94b611864b5d90f0e18d24801347c564eaffa7312c"} Oct 30 18:26:20.766670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:20.766495 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" event={"ID":"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd","Type":"ContainerStarted","Data":"7cce503b736d3e5352f8e901ef69d8fc7acb3e066bc2f3fb9a9df0d9e2529f98"} Oct 30 18:26:20.801645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:20.801594 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" podStartSLOduration=1.801576695 podStartE2EDuration="1.801576695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:20.799991318 +0000 UTC m=+1677.624385610" watchObservedRunningTime="2026-10-30 18:26:20.801576695 +0000 UTC m=+1677.625970988" Oct 30 18:26:21.366311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:21.366208 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:21.775034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:21.774999 4582 generic.go:356] "Generic (PLEG): container finished" podID="45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" containerID="cd7132eccf2952d0db631f94b611864b5d90f0e18d24801347c564eaffa7312c" exitCode=0 Oct 30 18:26:21.775517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:21.775080 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" event={"ID":"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd","Type":"ContainerDied","Data":"cd7132eccf2952d0db631f94b611864b5d90f0e18d24801347c564eaffa7312c"} Oct 30 18:26:22.932037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:22.932006 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:23.014972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.014935 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-kube-api-access-sx8pc\" (UniqueName: \"kubernetes.io/projected/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-kube-api-access-sx8pc\") pod \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " Oct 30 18:26:23.015133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.015025 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-xks-ca\") pod \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " Oct 30 18:26:23.015133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.015101 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-openshift-ca\") pod \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\" (UID: \"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd\") " Oct 30 18:26:23.015481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.015459 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-openshift-ca" pod "45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" (UID: "45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:23.017058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.017036 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-kube-api-access-sx8pc" pod "45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" (UID: "45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd"). InnerVolumeSpecName "kube-api-access-sx8pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:23.017058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.017047 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-xks-ca" pod "45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" (UID: "45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:23.055105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.055039 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:23.117379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.117345 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8pc\" (UniqueName: \"kubernetes.io/projected/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-kube-api-access-sx8pc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:23.117576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.117382 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:23.117576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.117402 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:23.209186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.209152 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44"] Oct 30 18:26:23.209390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.209367 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" containerID="cri-o://da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836" gracePeriod=30 Oct 30 18:26:23.212473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.212447 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt"] Oct 30 18:26:23.212706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.212641 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" containerID="cri-o://09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5" gracePeriod=30 Oct 30 18:26:23.792499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.792474 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" Oct 30 18:26:23.792499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.792485 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg" event={"ID":"45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd","Type":"ContainerDied","Data":"7cce503b736d3e5352f8e901ef69d8fc7acb3e066bc2f3fb9a9df0d9e2529f98"} Oct 30 18:26:23.792741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:23.792528 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cce503b736d3e5352f8e901ef69d8fc7acb3e066bc2f3fb9a9df0d9e2529f98" Oct 30 18:26:24.788180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:24.788093 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:24.877702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:24.877627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:25.013462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:25.013430 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:26.662407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:26.662370 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:28.382668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.382598 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:28.572745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.572723 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:26:28.576948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.576930 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:26:28.702214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702180 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-envoy\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.702402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702233 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-kube-api-access-m8jmm\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-kube-api-access-m8jmm\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.702402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702274 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-credential-socket\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.702402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702311 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-token\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-token\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.702402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702348 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77961a0e-9b90-46b5-8a93-3047caa3e29d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77961a0e-9b90-46b5-8a93-3047caa3e29d-istiod-ca-cert\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.702402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702391 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-envoy\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.702705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702493 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-token\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-token\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.702705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702526 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-credential-socket\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.702705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702557 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-credential-socket" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.702705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702604 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-data\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.702705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702654 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-socket\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702707 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-certs\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702736 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-data\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702779 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77961a0e-9b90-46b5-8a93-3047caa3e29d-istiod-ca-cert" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702834 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-podinfo\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702864 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-podinfo\") pod \"77961a0e-9b90-46b5-8a93-3047caa3e29d\" (UID: \"77961a0e-9b90-46b5-8a93-3047caa3e29d\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702870 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-credential-socket" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702908 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-socket\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.702935 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istiod-ca-cert\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703006 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-certs\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703046 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-kube-api-access-k8q7v\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-kube-api-access-k8q7v\") pod \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\" (UID: \"b2acf72b-22e4-4fee-91a3-b83fe9a4920e\") " Oct 30 18:26:28.703088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703087 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-data" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703112 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-socket" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703316 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-data" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703564 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-certs" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703617 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703635 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703650 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703664 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703678 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.703699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703692 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77961a0e-9b90-46b5-8a93-3047caa3e29d-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.704206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703883 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istiod-ca-cert" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:28.704206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.703912 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-certs" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.704206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.704157 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-socket" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.705457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.705429 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-podinfo" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:28.705578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.705508 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-envoy" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.705578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.705534 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-token" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.705578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.705566 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-envoy" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.706233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.706207 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-kube-api-access-k8q7v" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "kube-api-access-k8q7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.706341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.706252 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-podinfo" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:28.706407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.706339 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-token" pod "b2acf72b-22e4-4fee-91a3-b83fe9a4920e" (UID: "b2acf72b-22e4-4fee-91a3-b83fe9a4920e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.706816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.706797 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-kube-api-access-m8jmm" pod "77961a0e-9b90-46b5-8a93-3047caa3e29d" (UID: "77961a0e-9b90-46b5-8a93-3047caa3e29d"). InnerVolumeSpecName "kube-api-access-m8jmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.805266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805183 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805214 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805226 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805238 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805261 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805274 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8q7v\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-kube-api-access-k8q7v\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805286 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805299 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jmm\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-kube-api-access-m8jmm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805312 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77961a0e-9b90-46b5-8a93-3047caa3e29d-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805322 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805333 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2acf72b-22e4-4fee-91a3-b83fe9a4920e-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.805610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.805343 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77961a0e-9b90-46b5-8a93-3047caa3e29d-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.831788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.831751 4582 generic.go:356] "Generic (PLEG): container finished" podID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerID="da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836" exitCode=0 Oct 30 18:26:28.831980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.831799 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" Oct 30 18:26:28.831980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.831878 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" event={"ID":"77961a0e-9b90-46b5-8a93-3047caa3e29d","Type":"ContainerDied","Data":"da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836"} Oct 30 18:26:28.831980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.831942 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44" event={"ID":"77961a0e-9b90-46b5-8a93-3047caa3e29d","Type":"ContainerDied","Data":"81c1b0845fb75f5ea19466bebaf35a22695fe10a92483253269aa6b28680f990"} Oct 30 18:26:28.831980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.831976 4582 scope.go:122] "RemoveContainer" containerID="da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836" Oct 30 18:26:28.838067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.838013 4582 generic.go:356] "Generic (PLEG): container finished" podID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerID="09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5" exitCode=0 Oct 30 18:26:28.838067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.838060 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" Oct 30 18:26:28.838271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.838169 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" event={"ID":"b2acf72b-22e4-4fee-91a3-b83fe9a4920e","Type":"ContainerDied","Data":"09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5"} Oct 30 18:26:28.838271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.838230 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt" event={"ID":"b2acf72b-22e4-4fee-91a3-b83fe9a4920e","Type":"ContainerDied","Data":"773ede90a0a74e3a82eee9b6dfa32c04a918ae3870e0711a2d95d030ad5be68e"} Oct 30 18:26:28.851770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.851749 4582 scope.go:122] "RemoveContainer" containerID="da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836" Oct 30 18:26:28.852109 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:28.852084 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836\": container with ID starting with da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836 not found: ID does not exist" containerID="da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836" Oct 30 18:26:28.852189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.852118 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836"} err="failed to get container status \"da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836\": rpc error: code = NotFound desc = could not find container \"da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836\": container with ID starting with da5914883739064f2703bad6890ce389679327e2c5ebeae87a414a39bf62c836 not found: ID does not exist" Oct 30 18:26:28.852189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.852142 4582 scope.go:122] "RemoveContainer" containerID="09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5" Oct 30 18:26:28.862315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.862285 4582 scope.go:122] "RemoveContainer" containerID="09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5" Oct 30 18:26:28.862615 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:28.862594 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5\": container with ID starting with 09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5 not found: ID does not exist" containerID="09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5" Oct 30 18:26:28.862778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.862623 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5"} err="failed to get container status \"09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5\": rpc error: code = NotFound desc = could not find container \"09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5\": container with ID starting with 09110d05fed59988a3874e4f255fef5bc6f55f594b68dc1aff3476bc7c708cb5 not found: ID does not exist" Oct 30 18:26:28.890132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.890103 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44"] Oct 30 18:26:28.895194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.895168 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vsp44"] Oct 30 18:26:28.921779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.921757 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt"] Oct 30 18:26:28.935593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:28.935569 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-pssdt"] Oct 30 18:26:29.304987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:29.304939 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" path="/var/lib/kubelet/pods/77961a0e-9b90-46b5-8a93-3047caa3e29d/volumes" Oct 30 18:26:29.305671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:29.305651 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" path="/var/lib/kubelet/pods/b2acf72b-22e4-4fee-91a3-b83fe9a4920e/volumes" Oct 30 18:26:29.865748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:29.865716 4582 scope.go:122] "RemoveContainer" containerID="67d4dacc621e42f34074c3e9fcbdb83ca89503cdf60abc3d271b6a9c1c6ff399" Oct 30 18:26:30.481534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:30.481490 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:32.603456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:32.603424 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:35.126135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:35.126100 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:36.225167 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:36.225133 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:26:36.984537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:36.984498 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:37.906527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:37.906489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:38.997345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:38.997303 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:40.779419 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:40.779385 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:42.740443 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:42.740403 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:43.069460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:43.069382 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:44.802139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:44.802104 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:46.705339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:46.705305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:48.454712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:48.454673 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:49.829632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.829601 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268"] Oct 30 18:26:49.830176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830161 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" Oct 30 18:26:49.830227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830177 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" Oct 30 18:26:49.830227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830198 4582 cpu_manager.go:414] "containerMap: removing container" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" Oct 30 18:26:49.830227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830203 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" Oct 30 18:26:49.830227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830222 4582 cpu_manager.go:414] "containerMap: removing container" podUID="45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" containerName="revoke-keys" Oct 30 18:26:49.830227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830227 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" containerName="revoke-keys" Oct 30 18:26:49.830385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830320 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2acf72b-22e4-4fee-91a3-b83fe9a4920e" containerName="istio-proxy" Oct 30 18:26:49.830385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830333 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="77961a0e-9b90-46b5-8a93-3047caa3e29d" containerName="istio-proxy" Oct 30 18:26:49.830385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.830341 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" containerName="revoke-keys" Oct 30 18:26:49.833947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.833931 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:49.847885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.847864 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:49.851323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.851296 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268"] Oct 30 18:26:49.857324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.857299 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:26:49.908757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.908723 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:49.963626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.963596 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:49.963807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.963663 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcjf\" (UniqueName: \"kubernetes.io/projected/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-kube-api-access-2gcjf\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:49.963960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:49.963918 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.057758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.057718 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h"] Oct 30 18:26:50.058016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.057976 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" podUID="15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" containerName="payload-processing" containerID="cri-o://744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5" gracePeriod=30 Oct 30 18:26:50.065704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.065681 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.065887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.065741 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcjf\" (UniqueName: \"kubernetes.io/projected/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-kube-api-access-2gcjf\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.065955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.065916 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.066671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.066424 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.068220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.068198 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.079679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.079629 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84"] Oct 30 18:26:50.079845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.079801 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" podUID="4d0467aa-1432-4f65-92d9-85331482a312" containerName="payload-pre-processing" containerID="cri-o://db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68" gracePeriod=30 Oct 30 18:26:50.084014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.083811 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcjf\" (UniqueName: \"kubernetes.io/projected/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-kube-api-access-2gcjf\") pod \"maas-api-revoke-keys-e2e-praxis-a-50b278-n2268\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.140773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.140732 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:50.147322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.147295 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:50.239860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.237065 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:50.368433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.368409 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:26:50.449383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.447791 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:26:50.474607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.472598 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-plugins-config-volume\") pod \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " Oct 30 18:26:50.474607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.472753 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-kube-api-access-vsmrd\" (UniqueName: \"kubernetes.io/projected/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-kube-api-access-vsmrd\") pod \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\" (UID: \"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72\") " Oct 30 18:26:50.477848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.477800 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-plugins-config-volume" pod "15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" (UID: "15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:50.478021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.477978 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-kube-api-access-vsmrd" pod "15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" (UID: "15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72"). InnerVolumeSpecName "kube-api-access-vsmrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:50.480245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.480214 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268"] Oct 30 18:26:50.574640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.574599 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0467aa-1432-4f65-92d9-85331482a312-kube-api-access-ms4j4\" (UniqueName: \"kubernetes.io/projected/4d0467aa-1432-4f65-92d9-85331482a312-kube-api-access-ms4j4\") pod \"4d0467aa-1432-4f65-92d9-85331482a312\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " Oct 30 18:26:50.574847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.574682 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0467aa-1432-4f65-92d9-85331482a312-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0467aa-1432-4f65-92d9-85331482a312-plugins-config-volume\") pod \"4d0467aa-1432-4f65-92d9-85331482a312\" (UID: \"4d0467aa-1432-4f65-92d9-85331482a312\") " Oct 30 18:26:50.575092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.575067 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0467aa-1432-4f65-92d9-85331482a312-plugins-config-volume" pod "4d0467aa-1432-4f65-92d9-85331482a312" (UID: "4d0467aa-1432-4f65-92d9-85331482a312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:50.575203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.575187 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmrd\" (UniqueName: \"kubernetes.io/projected/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-kube-api-access-vsmrd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.575254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.575208 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0467aa-1432-4f65-92d9-85331482a312-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.575254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.575220 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.576633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.576609 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0467aa-1432-4f65-92d9-85331482a312-kube-api-access-ms4j4" pod "4d0467aa-1432-4f65-92d9-85331482a312" (UID: "4d0467aa-1432-4f65-92d9-85331482a312"). InnerVolumeSpecName "kube-api-access-ms4j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:50.676281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.676253 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4j4\" (UniqueName: \"kubernetes.io/projected/4d0467aa-1432-4f65-92d9-85331482a312-kube-api-access-ms4j4\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.714600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.714564 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:50.845033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.844995 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj"] Oct 30 18:26:50.854864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.854804 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f42594-6wnjj"] Oct 30 18:26:50.927087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.927003 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:50.987031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.986991 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" event={"ID":"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1","Type":"ContainerStarted","Data":"e951a4bba81ec6596bcac37270d2a041c38234cb0fbc4ba199fb9ac2e081c443"} Oct 30 18:26:50.987191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.987060 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" event={"ID":"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1","Type":"ContainerStarted","Data":"552551099fc74be21008cbc1dc78c01613f6751fbb0bcf365acfd26efdab8464"} Oct 30 18:26:50.988467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.988439 4582 generic.go:356] "Generic (PLEG): container finished" podID="15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" containerID="744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5" exitCode=0 Oct 30 18:26:50.988620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.988477 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" Oct 30 18:26:50.988620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.988521 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" event={"ID":"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72","Type":"ContainerDied","Data":"744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5"} Oct 30 18:26:50.988620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.988564 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h" event={"ID":"15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72","Type":"ContainerDied","Data":"111cb07b55dbe55f3ebc0b7cbcd70fd97139d19a75e4b61b3228489c7a874f24"} Oct 30 18:26:50.988620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.988586 4582 scope.go:122] "RemoveContainer" containerID="744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5" Oct 30 18:26:50.989994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.989971 4582 generic.go:356] "Generic (PLEG): container finished" podID="4d0467aa-1432-4f65-92d9-85331482a312" containerID="db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68" exitCode=0 Oct 30 18:26:50.990117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.990017 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" Oct 30 18:26:50.990117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.990059 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" event={"ID":"4d0467aa-1432-4f65-92d9-85331482a312","Type":"ContainerDied","Data":"db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68"} Oct 30 18:26:50.990252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:50.990116 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84" event={"ID":"4d0467aa-1432-4f65-92d9-85331482a312","Type":"ContainerDied","Data":"4e3d252d6b03b03331ace06e2f3feef78a5e41e0d70013c35ee23c122e4b4cdd"} Oct 30 18:26:51.004230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.000011 4582 scope.go:122] "RemoveContainer" containerID="744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5" Oct 30 18:26:51.004230 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:51.000345 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5\": container with ID starting with 744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5 not found: ID does not exist" containerID="744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5" Oct 30 18:26:51.004230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.000388 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5"} err="failed to get container status \"744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5\": rpc error: code = NotFound desc = could not find container \"744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5\": container with ID starting with 744ca1e90e6106518519c29da7254048b73832efb925fedebefab022730812f5 not found: ID does not exist" Oct 30 18:26:51.004230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.000415 4582 scope.go:122] "RemoveContainer" containerID="db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68" Oct 30 18:26:51.016482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.016463 4582 scope.go:122] "RemoveContainer" containerID="db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68" Oct 30 18:26:51.016776 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:26:51.016757 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68\": container with ID starting with db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68 not found: ID does not exist" containerID="db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68" Oct 30 18:26:51.016885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.016790 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68"} err="failed to get container status \"db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68\": rpc error: code = NotFound desc = could not find container \"db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68\": container with ID starting with db454d0417c110ef23cc570645bb796ed2a0ca58fb9a17b2b4706b7da2695b68 not found: ID does not exist" Oct 30 18:26:51.023340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.023287 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" podStartSLOduration=2.023271136 podStartE2EDuration="2.023271136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:51.02015822 +0000 UTC m=+1707.844552513" watchObservedRunningTime="2026-10-30 18:26:51.023271136 +0000 UTC m=+1707.847665428" Oct 30 18:26:51.051200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.051176 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84"] Oct 30 18:26:51.054598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.054579 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f42594-6fc446668c-6jr84"] Oct 30 18:26:51.083017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.082986 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h"] Oct 30 18:26:51.086028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.086008 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f42594-87777b966-2nn2h"] Oct 30 18:26:51.296397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.296362 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:51.300407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.300388 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" path="/var/lib/kubelet/pods/15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72/volumes" Oct 30 18:26:51.300777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.300764 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0467aa-1432-4f65-92d9-85331482a312" path="/var/lib/kubelet/pods/4d0467aa-1432-4f65-92d9-85331482a312/volumes" Oct 30 18:26:51.301186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:51.301165 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d14bb58-c0be-413e-b24b-c9cc3d8606d3" path="/var/lib/kubelet/pods/6d14bb58-c0be-413e-b24b-c9cc3d8606d3/volumes" Oct 30 18:26:52.002095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:52.002059 4582 generic.go:356] "Generic (PLEG): container finished" podID="4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" containerID="e951a4bba81ec6596bcac37270d2a041c38234cb0fbc4ba199fb9ac2e081c443" exitCode=0 Oct 30 18:26:52.002503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:52.002148 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" event={"ID":"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1","Type":"ContainerDied","Data":"e951a4bba81ec6596bcac37270d2a041c38234cb0fbc4ba199fb9ac2e081c443"} Oct 30 18:26:52.768876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:52.768845 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:53.158189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.158165 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:53.207420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.207387 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-kube-api-access-2gcjf\" (UniqueName: \"kubernetes.io/projected/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-kube-api-access-2gcjf\") pod \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " Oct 30 18:26:53.207613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.207439 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-xks-ca\") pod \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " Oct 30 18:26:53.207613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.207470 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-openshift-ca\") pod \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\" (UID: \"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1\") " Oct 30 18:26:53.208201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.208166 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-openshift-ca" pod "4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" (UID: "4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:53.209579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.209552 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-kube-api-access-2gcjf" pod "4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" (UID: "4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1"). InnerVolumeSpecName "kube-api-access-2gcjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:53.209987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.209959 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-xks-ca" pod "4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" (UID: "4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:53.309076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.309051 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcjf\" (UniqueName: \"kubernetes.io/projected/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-kube-api-access-2gcjf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:53.309201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.309078 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:53.309201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:53.309093 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:26:54.027123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.027094 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" Oct 30 18:26:54.027123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.027110 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268" event={"ID":"4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1","Type":"ContainerDied","Data":"552551099fc74be21008cbc1dc78c01613f6751fbb0bcf365acfd26efdab8464"} Oct 30 18:26:54.027337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.027179 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="552551099fc74be21008cbc1dc78c01613f6751fbb0bcf365acfd26efdab8464" Oct 30 18:26:54.283797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.283709 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:54.567782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.567694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:54.695400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.695369 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb"] Oct 30 18:26:54.695971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.695955 4582 cpu_manager.go:414] "containerMap: removing container" podUID="4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" containerName="revoke-keys" Oct 30 18:26:54.696020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.695972 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" containerName="revoke-keys" Oct 30 18:26:54.696020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.695986 4582 cpu_manager.go:414] "containerMap: removing container" podUID="15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" containerName="payload-processing" Oct 30 18:26:54.696020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.695991 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" containerName="payload-processing" Oct 30 18:26:54.696130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.696024 4582 cpu_manager.go:414] "containerMap: removing container" podUID="4d0467aa-1432-4f65-92d9-85331482a312" containerName="payload-pre-processing" Oct 30 18:26:54.696130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.696029 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0467aa-1432-4f65-92d9-85331482a312" containerName="payload-pre-processing" Oct 30 18:26:54.696130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.696111 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" containerName="revoke-keys" Oct 30 18:26:54.696130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.696118 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cbd5d9-3f82-4d7f-90a2-04d3b9cd1c72" containerName="payload-processing" Oct 30 18:26:54.696258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.696132 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0467aa-1432-4f65-92d9-85331482a312" containerName="payload-pre-processing" Oct 30 18:26:54.698664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.698637 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.712178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.711486 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-42bf6e-openshift-default-dockercfg-z44ww\"" Oct 30 18:26:54.728784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.728755 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb"] Oct 30 18:26:54.742526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.742496 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.832657 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istiod-ca-cert\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.847645 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-podinfo\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.847706 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-certs\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.847758 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-token\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.847791 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-credential-socket\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.847866 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-socket\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.847894 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-envoy\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.847957 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ckc\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-kube-api-access-t5ckc\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.848404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.848085 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-data\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.949724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.949690 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-socket\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.949724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.949723 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-envoy\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.949756 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ckc\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-kube-api-access-t5ckc\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.949861 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-data\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.949953 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istiod-ca-cert\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.949989 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-podinfo\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950012 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-certs\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950059 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-token\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950089 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-credential-socket\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950203 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-socket\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950280 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-data\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950330 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-certs\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950400 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-credential-socket\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.950610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.950592 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istiod-ca-cert\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.952140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.952113 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-envoy\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.952490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.952466 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-podinfo\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.966434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.966410 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-token\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:54.966543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:54.966526 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ckc\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-kube-api-access-t5ckc\") pod \"e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:55.021204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:55.021176 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:55.052929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:55.052902 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:55.336921 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:55.336891 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb"] Oct 30 18:26:55.340991 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:26:55.340955 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9defb28d_0725_409e_80d5_ba3b0df5d4fd.slice/crio-4d8aa523ccc70909c95e3a1bad356d5af4a756f8932bb0b3b0cd80460f3eff44 WatchSource:0}: Error finding container 4d8aa523ccc70909c95e3a1bad356d5af4a756f8932bb0b3b0cd80460f3eff44: Status 404 returned error can't find the container with id 4d8aa523ccc70909c95e3a1bad356d5af4a756f8932bb0b3b0cd80460f3eff44 Oct 30 18:26:55.343539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:55.343508 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:26:55.343621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:55.343572 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:26:55.343662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:55.343618 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:26:56.042731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:56.042686 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" event={"ID":"9defb28d-0725-409e-80d5-ba3b0df5d4fd","Type":"ContainerStarted","Data":"4ef30f742dcc2cd71f2c09724d0154063f9542b7730e02c11588f9d357efb16e"} Oct 30 18:26:56.042946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:56.042750 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" event={"ID":"9defb28d-0725-409e-80d5-ba3b0df5d4fd","Type":"ContainerStarted","Data":"4d8aa523ccc70909c95e3a1bad356d5af4a756f8932bb0b3b0cd80460f3eff44"} Oct 30 18:26:56.088599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:56.088547 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podStartSLOduration=2.088533276 podStartE2EDuration="2.088533276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:56.083121957 +0000 UTC m=+1712.907516249" watchObservedRunningTime="2026-10-30 18:26:56.088533276 +0000 UTC m=+1712.912927568" Oct 30 18:26:56.260683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:56.260648 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh"] Oct 30 18:26:56.320724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:56.320643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:56.395106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:56.395076 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:57.022101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:57.022070 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:57.028347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:57.028325 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:57.049495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:57.049468 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:57.050321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:57.050272 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:26:58.059242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.059212 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj"] Oct 30 18:26:58.093117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.093078 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx"] Oct 30 18:26:58.189717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.189635 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:58.288916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.288885 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:58.392661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.392615 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:58.489430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.489381 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:58.596541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.596489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:58.797455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:26:58.797362 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:00.069373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.069322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:00.172194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.172161 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:00.376001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.375968 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:00.477653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.477612 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:00.570515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.570474 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662"] Oct 30 18:27:00.574480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.574456 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.587156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.587124 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662"] Oct 30 18:27:00.593248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.593225 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-42bf6e\"" Oct 30 18:27:00.593383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.593280 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-42bf6e-dockercfg-mkn79\"" Oct 30 18:27:00.721885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.721782 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5c594e-4039-47db-9533-c5db3b98ddba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.722971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.722940 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2pb\" (UniqueName: \"kubernetes.io/projected/dc5c594e-4039-47db-9533-c5db3b98ddba-kube-api-access-mj2pb\") pod \"payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.730348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.730324 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr"] Oct 30 18:27:00.736766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.736745 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:00.770200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.770168 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr"] Oct 30 18:27:00.824928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.824896 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2pb\" (UniqueName: \"kubernetes.io/projected/dc5c594e-4039-47db-9533-c5db3b98ddba-kube-api-access-mj2pb\") pod \"payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.825080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.825046 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5c594e-4039-47db-9533-c5db3b98ddba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.825802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.825781 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5c594e-4039-47db-9533-c5db3b98ddba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.853557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.853520 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2pb\" (UniqueName: \"kubernetes.io/projected/dc5c594e-4039-47db-9533-c5db3b98ddba-kube-api-access-mj2pb\") pod \"payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.888480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.888450 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:00.926154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.926119 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef958611-b15d-4654-9144-1c15388dbef3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:00.926340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:00.926315 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87jcm\" (UniqueName: \"kubernetes.io/projected/ef958611-b15d-4654-9144-1c15388dbef3-kube-api-access-87jcm\") pod \"payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:01.028427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.028318 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef958611-b15d-4654-9144-1c15388dbef3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:01.028571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.028493 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jcm\" (UniqueName: \"kubernetes.io/projected/ef958611-b15d-4654-9144-1c15388dbef3-kube-api-access-87jcm\") pod \"payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:01.029125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.029105 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef958611-b15d-4654-9144-1c15388dbef3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:01.069007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.068974 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87jcm\" (UniqueName: \"kubernetes.io/projected/ef958611-b15d-4654-9144-1c15388dbef3-kube-api-access-87jcm\") pod \"payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:01.102288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.102258 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:01.213335 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.213305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:01.218506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.218434 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662"] Oct 30 18:27:01.221715 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:01.221688 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5c594e_4039_47db_9533_c5db3b98ddba.slice/crio-b8abe9bb900731f6f4d54b6dbf2e103abe7c9db4d2f229244d862dd4d7fd6df8 WatchSource:0}: Error finding container b8abe9bb900731f6f4d54b6dbf2e103abe7c9db4d2f229244d862dd4d7fd6df8: Status 404 returned error can't find the container with id b8abe9bb900731f6f4d54b6dbf2e103abe7c9db4d2f229244d862dd4d7fd6df8 Oct 30 18:27:01.430128 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.430094 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr"] Oct 30 18:27:01.433633 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:01.433605 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef958611_b15d_4654_9144_1c15388dbef3.slice/crio-af3c700fd0009b99d945e67907766323f2256d866eac6802e41a0951b84fa3c0 WatchSource:0}: Error finding container af3c700fd0009b99d945e67907766323f2256d866eac6802e41a0951b84fa3c0: Status 404 returned error can't find the container with id af3c700fd0009b99d945e67907766323f2256d866eac6802e41a0951b84fa3c0 Oct 30 18:27:01.836897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.836857 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:01.934424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:01.934388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:02.092371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.092286 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" event={"ID":"dc5c594e-4039-47db-9533-c5db3b98ddba","Type":"ContainerStarted","Data":"3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a"} Oct 30 18:27:02.092371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.092342 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" event={"ID":"dc5c594e-4039-47db-9533-c5db3b98ddba","Type":"ContainerStarted","Data":"b8abe9bb900731f6f4d54b6dbf2e103abe7c9db4d2f229244d862dd4d7fd6df8"} Oct 30 18:27:02.092613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.092377 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:02.094077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.094054 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" event={"ID":"ef958611-b15d-4654-9144-1c15388dbef3","Type":"ContainerStarted","Data":"29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf"} Oct 30 18:27:02.094213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.094088 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" event={"ID":"ef958611-b15d-4654-9144-1c15388dbef3","Type":"ContainerStarted","Data":"af3c700fd0009b99d945e67907766323f2256d866eac6802e41a0951b84fa3c0"} Oct 30 18:27:02.094213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.094141 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:02.126905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.126862 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" podStartSLOduration=2.126846505 podStartE2EDuration="2.126846505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:02.124172919 +0000 UTC m=+1718.948567222" watchObservedRunningTime="2026-10-30 18:27:02.126846505 +0000 UTC m=+1718.951240805" Oct 30 18:27:02.965844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:02.963765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:03.067974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.067898 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" podUID="89cacc01-2b2b-496c-aa16-4a5f92fbca16" containerName="payload-processing" containerID="cri-o://e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a" gracePeriod=25 Oct 30 18:27:03.093657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.093625 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" podUID="38246359-a129-4c55-bccc-3979b7baf132" containerName="payload-pre-processing" containerID="cri-o://19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88" gracePeriod=25 Oct 30 18:27:03.423540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.423513 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:27:03.438763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.438737 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:27:03.468870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.468770 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" podStartSLOduration=3.468751227 podStartE2EDuration="3.468751227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:02.165184232 +0000 UTC m=+1718.989578551" watchObservedRunningTime="2026-10-30 18:27:03.468751227 +0000 UTC m=+1720.293145520" Oct 30 18:27:03.492761 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.492731 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cacc01-2b2b-496c-aa16-4a5f92fbca16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cacc01-2b2b-496c-aa16-4a5f92fbca16-plugins-config-volume\") pod \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " Oct 30 18:27:03.492968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.492816 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38246359-a129-4c55-bccc-3979b7baf132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38246359-a129-4c55-bccc-3979b7baf132-plugins-config-volume\") pod \"38246359-a129-4c55-bccc-3979b7baf132\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " Oct 30 18:27:03.492968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.492883 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38246359-a129-4c55-bccc-3979b7baf132-kube-api-access-z86f9\" (UniqueName: \"kubernetes.io/projected/38246359-a129-4c55-bccc-3979b7baf132-kube-api-access-z86f9\") pod \"38246359-a129-4c55-bccc-3979b7baf132\" (UID: \"38246359-a129-4c55-bccc-3979b7baf132\") " Oct 30 18:27:03.492968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.492929 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cacc01-2b2b-496c-aa16-4a5f92fbca16-kube-api-access-9rtg2\" (UniqueName: \"kubernetes.io/projected/89cacc01-2b2b-496c-aa16-4a5f92fbca16-kube-api-access-9rtg2\") pod \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\" (UID: \"89cacc01-2b2b-496c-aa16-4a5f92fbca16\") " Oct 30 18:27:03.493221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.493193 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cacc01-2b2b-496c-aa16-4a5f92fbca16-plugins-config-volume" pod "89cacc01-2b2b-496c-aa16-4a5f92fbca16" (UID: "89cacc01-2b2b-496c-aa16-4a5f92fbca16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:03.493319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.493294 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38246359-a129-4c55-bccc-3979b7baf132-plugins-config-volume" pod "38246359-a129-4c55-bccc-3979b7baf132" (UID: "38246359-a129-4c55-bccc-3979b7baf132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:03.493630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.493596 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cacc01-2b2b-496c-aa16-4a5f92fbca16-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:03.493630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.493625 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38246359-a129-4c55-bccc-3979b7baf132-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:03.495471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.495449 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38246359-a129-4c55-bccc-3979b7baf132-kube-api-access-z86f9" pod "38246359-a129-4c55-bccc-3979b7baf132" (UID: "38246359-a129-4c55-bccc-3979b7baf132"). InnerVolumeSpecName "kube-api-access-z86f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:03.495587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.495545 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cacc01-2b2b-496c-aa16-4a5f92fbca16-kube-api-access-9rtg2" pod "89cacc01-2b2b-496c-aa16-4a5f92fbca16" (UID: "89cacc01-2b2b-496c-aa16-4a5f92fbca16"). InnerVolumeSpecName "kube-api-access-9rtg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:03.595262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.595183 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z86f9\" (UniqueName: \"kubernetes.io/projected/38246359-a129-4c55-bccc-3979b7baf132-kube-api-access-z86f9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:03.595262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:03.595211 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtg2\" (UniqueName: \"kubernetes.io/projected/89cacc01-2b2b-496c-aa16-4a5f92fbca16-kube-api-access-9rtg2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.109141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.109102 4582 generic.go:356] "Generic (PLEG): container finished" podID="38246359-a129-4c55-bccc-3979b7baf132" containerID="19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88" exitCode=0 Oct 30 18:27:04.109321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.109148 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" Oct 30 18:27:04.109321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.109190 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" event={"ID":"38246359-a129-4c55-bccc-3979b7baf132","Type":"ContainerDied","Data":"19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88"} Oct 30 18:27:04.109321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.109239 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx" event={"ID":"38246359-a129-4c55-bccc-3979b7baf132","Type":"ContainerDied","Data":"efe34249c91665f5f1516aa4df8d2621cfffd87a23f3b964c3d891c446591621"} Oct 30 18:27:04.109321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.109261 4582 scope.go:122] "RemoveContainer" containerID="19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88" Oct 30 18:27:04.110617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.110544 4582 generic.go:356] "Generic (PLEG): container finished" podID="89cacc01-2b2b-496c-aa16-4a5f92fbca16" containerID="e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a" exitCode=0 Oct 30 18:27:04.110617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.110588 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" Oct 30 18:27:04.110617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.110609 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" event={"ID":"89cacc01-2b2b-496c-aa16-4a5f92fbca16","Type":"ContainerDied","Data":"e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a"} Oct 30 18:27:04.110747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.110651 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj" event={"ID":"89cacc01-2b2b-496c-aa16-4a5f92fbca16","Type":"ContainerDied","Data":"07d19210fdb0a930e974d91a9d4360503304369c4e8b2e1315920d2860b15ac4"} Oct 30 18:27:04.121596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.121577 4582 scope.go:122] "RemoveContainer" containerID="19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88" Oct 30 18:27:04.122008 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:04.121894 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88\": container with ID starting with 19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88 not found: ID does not exist" containerID="19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88" Oct 30 18:27:04.122008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.121929 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88"} err="failed to get container status \"19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88\": rpc error: code = NotFound desc = could not find container \"19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88\": container with ID starting with 19ac48ffce29d3ba009b7bd704e64ef763c3389d84ad918aca42a10670925b88 not found: ID does not exist" Oct 30 18:27:04.122008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.121950 4582 scope.go:122] "RemoveContainer" containerID="e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a" Oct 30 18:27:04.130783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.130768 4582 scope.go:122] "RemoveContainer" containerID="e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a" Oct 30 18:27:04.131080 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:04.131060 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a\": container with ID starting with e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a not found: ID does not exist" containerID="e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a" Oct 30 18:27:04.131141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.131088 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a"} err="failed to get container status \"e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a\": rpc error: code = NotFound desc = could not find container \"e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a\": container with ID starting with e7a13c3b98ca1c05536c9375d9d637b44b4edd8757ff3be9957afee4713a299a not found: ID does not exist" Oct 30 18:27:04.176429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.176396 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj"] Oct 30 18:27:04.185815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.185788 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-6f5c4477d6-wd4nj"] Oct 30 18:27:04.211082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.211057 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx"] Oct 30 18:27:04.217286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.217263 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-f5b46dc95-hh5lx"] Oct 30 18:27:04.594254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:04.594214 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:05.307116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:05.307026 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38246359-a129-4c55-bccc-3979b7baf132" path="/var/lib/kubelet/pods/38246359-a129-4c55-bccc-3979b7baf132/volumes" Oct 30 18:27:05.307538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:05.307519 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cacc01-2b2b-496c-aa16-4a5f92fbca16" path="/var/lib/kubelet/pods/89cacc01-2b2b-496c-aa16-4a5f92fbca16/volumes" Oct 30 18:27:06.226140 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:06.226103 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:27:06.317237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:06.317200 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:08.083442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:08.083395 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:09.162261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:09.162227 4582 generic.go:356] "Generic (PLEG): container finished" podID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerID="8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893" exitCode=0 Oct 30 18:27:09.162869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:09.162273 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" event={"ID":"50651ef2-6ef6-41d3-ab69-893124fb312b","Type":"ContainerDied","Data":"8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893"} Oct 30 18:27:09.162869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:09.162798 4582 scope.go:122] "RemoveContainer" containerID="8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893" Oct 30 18:27:09.164997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:09.164967 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:27:09.165095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:09.165046 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:27:09.165095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:09.165078 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:27:09.797489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:09.797458 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:10.175738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:10.175702 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" event={"ID":"50651ef2-6ef6-41d3-ab69-893124fb312b","Type":"ContainerStarted","Data":"7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0"} Oct 30 18:27:10.699705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:10.699661 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:27:10.699966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:10.699814 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:27:10.699966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:10.699914 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 18:27:10.782771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:10.782738 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:11.281624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.281556 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" podUID="5aa8d637-de56-45b3-aae9-1ce1da124411" containerName="main" containerID="cri-o://c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a" gracePeriod=45 Oct 30 18:27:11.295729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.295682 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:11.392865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.392797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:11.637772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.637740 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:27:11.713992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.713353 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa8d637-de56-45b3-aae9-1ce1da124411-tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa8d637-de56-45b3-aae9-1ce1da124411-tls-certs\") pod \"5aa8d637-de56-45b3-aae9-1ce1da124411\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " Oct 30 18:27:11.713992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.713416 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa8d637-de56-45b3-aae9-1ce1da124411-kube-api-access-4jq98\" (UniqueName: \"kubernetes.io/projected/5aa8d637-de56-45b3-aae9-1ce1da124411-kube-api-access-4jq98\") pod \"5aa8d637-de56-45b3-aae9-1ce1da124411\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " Oct 30 18:27:11.713992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.713463 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-tmp-dir\") pod \"5aa8d637-de56-45b3-aae9-1ce1da124411\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " Oct 30 18:27:11.713992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.713632 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-model-cache\") pod \"5aa8d637-de56-45b3-aae9-1ce1da124411\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " Oct 30 18:27:11.714616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.714407 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-model-cache" pod "5aa8d637-de56-45b3-aae9-1ce1da124411" (UID: "5aa8d637-de56-45b3-aae9-1ce1da124411"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:11.719857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.714771 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:27:11.719857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.714838 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 18:27:11.720136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.720093 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-tmp-dir" pod "5aa8d637-de56-45b3-aae9-1ce1da124411" (UID: "5aa8d637-de56-45b3-aae9-1ce1da124411"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:11.729840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.720247 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-dshm\") pod \"5aa8d637-de56-45b3-aae9-1ce1da124411\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " Oct 30 18:27:11.729840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.720319 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-home\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-home\") pod \"5aa8d637-de56-45b3-aae9-1ce1da124411\" (UID: \"5aa8d637-de56-45b3-aae9-1ce1da124411\") " Oct 30 18:27:11.729840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.722201 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-home" pod "5aa8d637-de56-45b3-aae9-1ce1da124411" (UID: "5aa8d637-de56-45b3-aae9-1ce1da124411"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:11.730461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.730402 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.730461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.730428 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.730461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.730441 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.740272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.740231 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa8d637-de56-45b3-aae9-1ce1da124411-kube-api-access-4jq98" pod "5aa8d637-de56-45b3-aae9-1ce1da124411" (UID: "5aa8d637-de56-45b3-aae9-1ce1da124411"). InnerVolumeSpecName "kube-api-access-4jq98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:11.788187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.788150 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-dshm" pod "5aa8d637-de56-45b3-aae9-1ce1da124411" (UID: "5aa8d637-de56-45b3-aae9-1ce1da124411"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:11.793196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.792977 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa8d637-de56-45b3-aae9-1ce1da124411-tls-certs" pod "5aa8d637-de56-45b3-aae9-1ce1da124411" (UID: "5aa8d637-de56-45b3-aae9-1ce1da124411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:11.831781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.831755 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa8d637-de56-45b3-aae9-1ce1da124411-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.831781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.831777 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa8d637-de56-45b3-aae9-1ce1da124411-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.831781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.831788 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jq98\" (UniqueName: \"kubernetes.io/projected/5aa8d637-de56-45b3-aae9-1ce1da124411-kube-api-access-4jq98\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:11.895895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:11.895865 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:12.063856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.059536 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:12.106477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.106369 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268"] Oct 30 18:27:12.110100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.110060 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-50b278-n2268"] Oct 30 18:27:12.168201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.168115 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:12.190811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.190779 4582 generic.go:356] "Generic (PLEG): container finished" podID="5aa8d637-de56-45b3-aae9-1ce1da124411" containerID="c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a" exitCode=0 Oct 30 18:27:12.190811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.190832 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" Oct 30 18:27:12.191018 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.190853 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" event={"ID":"5aa8d637-de56-45b3-aae9-1ce1da124411","Type":"ContainerDied","Data":"c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a"} Oct 30 18:27:12.191018 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.190888 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh" event={"ID":"5aa8d637-de56-45b3-aae9-1ce1da124411","Type":"ContainerDied","Data":"c35447f8bdb5f370c15469d31f2f2ebcd85f0697b10949732cf549442939f96c"} Oct 30 18:27:12.191018 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.190908 4582 scope.go:122] "RemoveContainer" containerID="c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a" Oct 30 18:27:12.201042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.201025 4582 scope.go:122] "RemoveContainer" containerID="c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a" Oct 30 18:27:12.201304 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:12.201283 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a\": container with ID starting with c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a not found: ID does not exist" containerID="c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a" Oct 30 18:27:12.201380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.201313 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a"} err="failed to get container status \"c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a\": rpc error: code = NotFound desc = could not find container \"c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a\": container with ID starting with c144e9c48c261ad377436070d95e6231dc4de75cf86ab5dacc7971c62257195a not found: ID does not exist" Oct 30 18:27:12.234328 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.234295 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh"] Oct 30 18:27:12.238127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.238100 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f42594/ipp-route-f42594-kserve-db64f4f49-w7dwh"] Oct 30 18:27:12.699841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.699785 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:27:12.700249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:12.699864 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 18:27:13.104373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:13.104295 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:27:13.104373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:13.104363 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:27:13.318845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:13.304589 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1" path="/var/lib/kubelet/pods/4a0edfb7-73e6-478b-a7bf-0abbf2eb1ce1/volumes" Oct 30 18:27:13.318845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:13.305164 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa8d637-de56-45b3-aae9-1ce1da124411" path="/var/lib/kubelet/pods/5aa8d637-de56-45b3-aae9-1ce1da124411/volumes" Oct 30 18:27:13.699972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:13.699937 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:27:13.700385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:13.700003 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 18:27:14.308670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:14.308638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:14.683147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:14.683118 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:14.699884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:14.699857 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:27:14.700036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:14.699907 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 18:27:15.601863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:15.601813 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:15.704708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:15.704678 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:27:15.705197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:15.705037 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:27:15.705601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:15.705583 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:27:15.772942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:15.771343 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:16.186428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.186386 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb"] Oct 30 18:27:16.187368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187339 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5aa8d637-de56-45b3-aae9-1ce1da124411" containerName="main" Oct 30 18:27:16.187368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187368 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa8d637-de56-45b3-aae9-1ce1da124411" containerName="main" Oct 30 18:27:16.187586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187396 4582 cpu_manager.go:414] "containerMap: removing container" podUID="89cacc01-2b2b-496c-aa16-4a5f92fbca16" containerName="payload-processing" Oct 30 18:27:16.187586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187404 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cacc01-2b2b-496c-aa16-4a5f92fbca16" containerName="payload-processing" Oct 30 18:27:16.187586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187443 4582 cpu_manager.go:414] "containerMap: removing container" podUID="38246359-a129-4c55-bccc-3979b7baf132" containerName="payload-pre-processing" Oct 30 18:27:16.187586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187452 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38246359-a129-4c55-bccc-3979b7baf132" containerName="payload-pre-processing" Oct 30 18:27:16.187804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187641 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa8d637-de56-45b3-aae9-1ce1da124411" containerName="main" Oct 30 18:27:16.187804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187655 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cacc01-2b2b-496c-aa16-4a5f92fbca16" containerName="payload-processing" Oct 30 18:27:16.187804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.187671 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="38246359-a129-4c55-bccc-3979b7baf132" containerName="payload-pre-processing" Oct 30 18:27:16.191715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.191690 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.208842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.207974 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb"] Oct 30 18:27:16.208842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.208015 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e9b146-dockercfg-rmcs5\"" Oct 30 18:27:16.213330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.213307 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e9b146\"" Oct 30 18:27:16.240676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.240648 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.240920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.240901 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbhk\" (UniqueName: \"kubernetes.io/projected/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-kube-api-access-6sbhk\") pod \"payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.342344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.342307 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.342480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.342435 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbhk\" (UniqueName: \"kubernetes.io/projected/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-kube-api-access-6sbhk\") pod \"payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.343026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.343005 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.346956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.346933 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x"] Oct 30 18:27:16.350223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.350204 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.361264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.361240 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbhk\" (UniqueName: \"kubernetes.io/projected/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-kube-api-access-6sbhk\") pod \"payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.371090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.371068 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x"] Oct 30 18:27:16.443899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.443809 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.444047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.444022 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjlk\" (UniqueName: \"kubernetes.io/projected/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-kube-api-access-xmjlk\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.516253 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.516219 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:16.545630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.545602 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.545758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.545702 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjlk\" (UniqueName: \"kubernetes.io/projected/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-kube-api-access-xmjlk\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.546308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.546283 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.562447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.562417 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjlk\" (UniqueName: \"kubernetes.io/projected/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-kube-api-access-xmjlk\") pod \"payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.662402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.662375 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:16.768251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.768203 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:16.810990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.810969 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb"] Oct 30 18:27:16.813361 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:16.813337 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf92986b_c01f_4f9d_8ac1_ce6d5eaf7660.slice/crio-088a7a5f8d80d00d423ae74c64d6ad453fa886a6648d4b0f1922a33f373fab42 WatchSource:0}: Error finding container 088a7a5f8d80d00d423ae74c64d6ad453fa886a6648d4b0f1922a33f373fab42: Status 404 returned error can't find the container with id 088a7a5f8d80d00d423ae74c64d6ad453fa886a6648d4b0f1922a33f373fab42 Oct 30 18:27:16.961167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:16.961136 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x"] Oct 30 18:27:16.964695 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:16.964645 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdd259e_80c6_4cea_855a_50e7f4ae4a66.slice/crio-70e436a35a9706abbb3065093ac706320108281f36ffdd30d6ac10377ff403c8 WatchSource:0}: Error finding container 70e436a35a9706abbb3065093ac706320108281f36ffdd30d6ac10377ff403c8: Status 404 returned error can't find the container with id 70e436a35a9706abbb3065093ac706320108281f36ffdd30d6ac10377ff403c8 Oct 30 18:27:17.235071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.234975 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" event={"ID":"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660","Type":"ContainerStarted","Data":"479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f"} Oct 30 18:27:17.235239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.235074 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" event={"ID":"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660","Type":"ContainerStarted","Data":"088a7a5f8d80d00d423ae74c64d6ad453fa886a6648d4b0f1922a33f373fab42"} Oct 30 18:27:17.235239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.235216 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:17.236857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.236810 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" event={"ID":"8bdd259e-80c6-4cea-855a-50e7f4ae4a66","Type":"ContainerStarted","Data":"0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4"} Oct 30 18:27:17.236946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.236872 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" event={"ID":"8bdd259e-80c6-4cea-855a-50e7f4ae4a66","Type":"ContainerStarted","Data":"70e436a35a9706abbb3065093ac706320108281f36ffdd30d6ac10377ff403c8"} Oct 30 18:27:17.236946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.236897 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:17.270651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.270605 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" podStartSLOduration=1.270594576 podStartE2EDuration="1.270594576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:17.270206156 +0000 UTC m=+1734.094600460" watchObservedRunningTime="2026-10-30 18:27:17.270594576 +0000 UTC m=+1734.094988867" Oct 30 18:27:17.305255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:17.305199 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" podStartSLOduration=1.30518285 podStartE2EDuration="1.30518285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:17.302710438 +0000 UTC m=+1734.127104750" watchObservedRunningTime="2026-10-30 18:27:17.30518285 +0000 UTC m=+1734.129577145" Oct 30 18:27:18.504956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:18.504919 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:20.184773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:20.184736 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:21.753409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.753375 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67"] Oct 30 18:27:21.756778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.756761 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.768433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.768408 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-42bf6e-serving-cert\"" Oct 30 18:27:21.786175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.786148 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67"] Oct 30 18:27:21.815672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.815638 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-metrics-certs\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.815858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.815802 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.815936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.815903 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.816012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.815996 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znsvn\" (UniqueName: \"kubernetes.io/projected/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-kube-api-access-znsvn\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.822339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.822295 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:21.904241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.904191 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:21.917577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.917547 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znsvn\" (UniqueName: \"kubernetes.io/projected/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-kube-api-access-znsvn\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.917718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.917643 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-metrics-certs\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.917718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.917712 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.917868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.917737 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.918406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.918376 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.920215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.920198 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.920287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.920260 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-metrics-certs\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:21.935907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:21.935886 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znsvn\" (UniqueName: \"kubernetes.io/projected/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-kube-api-access-znsvn\") pod \"maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:22.004847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:22.004735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:22.071234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:22.071207 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:22.369964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:22.369924 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67"] Oct 30 18:27:22.373679 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:22.373656 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c93c5c8_d95e_4dc7_ac89_8e5c87ce42f0.slice/crio-527d9e9c2ec5995fbacae4d71ce7adcead958394fbfda8844d3555dd8260465c WatchSource:0}: Error finding container 527d9e9c2ec5995fbacae4d71ce7adcead958394fbfda8844d3555dd8260465c: Status 404 returned error can't find the container with id 527d9e9c2ec5995fbacae4d71ce7adcead958394fbfda8844d3555dd8260465c Oct 30 18:27:23.282565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:23.282522 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" event={"ID":"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0","Type":"ContainerStarted","Data":"53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b"} Oct 30 18:27:23.282967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:23.282583 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" event={"ID":"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0","Type":"ContainerStarted","Data":"527d9e9c2ec5995fbacae4d71ce7adcead958394fbfda8844d3555dd8260465c"} Oct 30 18:27:23.282967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:23.282611 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:23.319939 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:23.319873 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" podStartSLOduration=1.902476429 podStartE2EDuration="2.319859219s" totalImagesPullingTime="417.38279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:23.317373012 +0000 UTC m=+1740.141767317" watchObservedRunningTime="2026-10-30 18:27:23.319859219 +0000 UTC m=+1740.144253533" Oct 30 18:27:23.878148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:23.878112 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:26.166141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:26.166106 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.954551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:27.954521 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:28.246295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:28.246215 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:27:28.246295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:28.246281 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:27:29.300205 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.300179 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:27:29.671550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.671517 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:29.918530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.918495 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2"] Oct 30 18:27:29.921585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.921540 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:29.932322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.932297 4582 scope.go:122] "RemoveContainer" containerID="baac27e5e42fcf700b7f89f34ebed62fe55f884ee850d68f7d2588a9a7f7de18" Oct 30 18:27:29.933311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.933291 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:29.942944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.942906 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:27:29.948920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:29.948895 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2"] Oct 30 18:27:30.016841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.016789 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.016943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.016875 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.017096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.017075 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dvw\" (UniqueName: \"kubernetes.io/projected/3b93d5d5-ca47-42a3-b018-ba1908b65eba-kube-api-access-h2dvw\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.118245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.118214 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.118402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.118263 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.118402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.118363 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dvw\" (UniqueName: \"kubernetes.io/projected/3b93d5d5-ca47-42a3-b018-ba1908b65eba-kube-api-access-h2dvw\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.119010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.118989 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.120502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.120481 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.134332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.134308 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dvw\" (UniqueName: \"kubernetes.io/projected/3b93d5d5-ca47-42a3-b018-ba1908b65eba-kube-api-access-h2dvw\") pod \"maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.225349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.225290 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:30.235235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.235212 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:30.541612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.541536 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2"] Oct 30 18:27:30.545011 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:30.544972 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b93d5d5_ca47_42a3_b018_ba1908b65eba.slice/crio-b387823a3d4fd1bb6eed9867319a20655435edd5a9bdd142edb15dc4f996437d WatchSource:0}: Error finding container b387823a3d4fd1bb6eed9867319a20655435edd5a9bdd142edb15dc4f996437d: Status 404 returned error can't find the container with id b387823a3d4fd1bb6eed9867319a20655435edd5a9bdd142edb15dc4f996437d Oct 30 18:27:30.690726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.690684 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:30.791414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:30.791384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:31.340780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:31.340699 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" event={"ID":"3b93d5d5-ca47-42a3-b018-ba1908b65eba","Type":"ContainerStarted","Data":"baddd0a0af25810c040c9a628b2e54c513750c755e0f223eec4e14ef9e48090e"} Oct 30 18:27:31.340780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:31.340764 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" event={"ID":"3b93d5d5-ca47-42a3-b018-ba1908b65eba","Type":"ContainerStarted","Data":"b387823a3d4fd1bb6eed9867319a20655435edd5a9bdd142edb15dc4f996437d"} Oct 30 18:27:31.376706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:31.376650 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" podStartSLOduration=2.376634786 podStartE2EDuration="2.376634786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:31.37602836 +0000 UTC m=+1748.200422654" watchObservedRunningTime="2026-10-30 18:27:31.376634786 +0000 UTC m=+1748.201029081" Oct 30 18:27:31.760143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:31.760109 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:32.349089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:32.349056 4582 generic.go:356] "Generic (PLEG): container finished" podID="3b93d5d5-ca47-42a3-b018-ba1908b65eba" containerID="baddd0a0af25810c040c9a628b2e54c513750c755e0f223eec4e14ef9e48090e" exitCode=0 Oct 30 18:27:32.349275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:32.349137 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" event={"ID":"3b93d5d5-ca47-42a3-b018-ba1908b65eba","Type":"ContainerDied","Data":"baddd0a0af25810c040c9a628b2e54c513750c755e0f223eec4e14ef9e48090e"} Oct 30 18:27:32.521100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:32.521064 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:32.621425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:32.621397 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:33.509884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.509860 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:33.640784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.640746 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:33.668218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.668189 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-openshift-ca\") pod \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " Oct 30 18:27:33.668364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.668221 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b93d5d5-ca47-42a3-b018-ba1908b65eba-kube-api-access-h2dvw\" (UniqueName: \"kubernetes.io/projected/3b93d5d5-ca47-42a3-b018-ba1908b65eba-kube-api-access-h2dvw\") pod \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " Oct 30 18:27:33.668364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.668272 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-xks-ca\") pod \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\" (UID: \"3b93d5d5-ca47-42a3-b018-ba1908b65eba\") " Oct 30 18:27:33.668607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.668579 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-openshift-ca" pod "3b93d5d5-ca47-42a3-b018-ba1908b65eba" (UID: "3b93d5d5-ca47-42a3-b018-ba1908b65eba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:33.668796 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.668775 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.670263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.670240 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-xks-ca" pod "3b93d5d5-ca47-42a3-b018-ba1908b65eba" (UID: "3b93d5d5-ca47-42a3-b018-ba1908b65eba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:33.670371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.670353 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b93d5d5-ca47-42a3-b018-ba1908b65eba-kube-api-access-h2dvw" pod "3b93d5d5-ca47-42a3-b018-ba1908b65eba" (UID: "3b93d5d5-ca47-42a3-b018-ba1908b65eba"). InnerVolumeSpecName "kube-api-access-h2dvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:33.769547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.769514 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dvw\" (UniqueName: \"kubernetes.io/projected/3b93d5d5-ca47-42a3-b018-ba1908b65eba-kube-api-access-h2dvw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.769547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:33.769542 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b93d5d5-ca47-42a3-b018-ba1908b65eba-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:34.136473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:34.136437 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:34.365032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:34.365001 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" event={"ID":"3b93d5d5-ca47-42a3-b018-ba1908b65eba","Type":"ContainerDied","Data":"b387823a3d4fd1bb6eed9867319a20655435edd5a9bdd142edb15dc4f996437d"} Oct 30 18:27:34.365032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:34.365026 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2" Oct 30 18:27:34.365246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:34.365047 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b387823a3d4fd1bb6eed9867319a20655435edd5a9bdd142edb15dc4f996437d" Oct 30 18:27:34.674835 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:34.674793 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:34.773501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:34.773465 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:35.515036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:35.514980 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:36.225318 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:36.225280 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:27:36.449065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.449016 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:36.538792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.538693 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:36.566142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.566119 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl"] Oct 30 18:27:36.566700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.566687 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3b93d5d5-ca47-42a3-b018-ba1908b65eba" containerName="revoke-keys" Oct 30 18:27:36.566741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.566701 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b93d5d5-ca47-42a3-b018-ba1908b65eba" containerName="revoke-keys" Oct 30 18:27:36.566850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.566840 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b93d5d5-ca47-42a3-b018-ba1908b65eba" containerName="revoke-keys" Oct 30 18:27:36.575145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.575112 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.588793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.588770 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl"] Oct 30 18:27:36.594846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.594803 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-44379a\"" Oct 30 18:27:36.594954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.594884 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-44379a-dockercfg-qz8z5\"" Oct 30 18:27:36.635477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.635422 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7"] Oct 30 18:27:36.639630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.639606 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:36.670139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.670087 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7"] Oct 30 18:27:36.703634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.703524 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac44c39c-548c-4ab7-a6f2-8f726b047d38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.703874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.703802 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgqw\" (UniqueName: \"kubernetes.io/projected/ac44c39c-548c-4ab7-a6f2-8f726b047d38-kube-api-access-xqgqw\") pod \"payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.805750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.805659 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgqw\" (UniqueName: \"kubernetes.io/projected/ac44c39c-548c-4ab7-a6f2-8f726b047d38-kube-api-access-xqgqw\") pod \"payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.805750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.805716 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac44c39c-548c-4ab7-a6f2-8f726b047d38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.806033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.805803 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/32d336ca-3ee7-42d3-855a-a06662e15ebc-kube-api-access-gvp6m\") pod \"payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:36.806033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.805934 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d336ca-3ee7-42d3-855a-a06662e15ebc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:36.806558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.806534 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac44c39c-548c-4ab7-a6f2-8f726b047d38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.822153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.822127 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgqw\" (UniqueName: \"kubernetes.io/projected/ac44c39c-548c-4ab7-a6f2-8f726b047d38-kube-api-access-xqgqw\") pod \"payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.889710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.889679 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:36.908017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.907986 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/32d336ca-3ee7-42d3-855a-a06662e15ebc-kube-api-access-gvp6m\") pod \"payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:36.908127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.908087 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d336ca-3ee7-42d3-855a-a06662e15ebc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:36.908762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.908741 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d336ca-3ee7-42d3-855a-a06662e15ebc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:36.925426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.925400 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/32d336ca-3ee7-42d3-855a-a06662e15ebc-kube-api-access-gvp6m\") pod \"payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:36.960323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:36.960085 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:37.187848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.187800 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl"] Oct 30 18:27:37.191864 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:37.191834 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac44c39c_548c_4ab7_a6f2_8f726b047d38.slice/crio-cddbcb6f87e60f74e207666a0abd6c38f6975584ad7632fb2e665253bd0fa438 WatchSource:0}: Error finding container cddbcb6f87e60f74e207666a0abd6c38f6975584ad7632fb2e665253bd0fa438: Status 404 returned error can't find the container with id cddbcb6f87e60f74e207666a0abd6c38f6975584ad7632fb2e665253bd0fa438 Oct 30 18:27:37.257451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.257427 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7"] Oct 30 18:27:37.261801 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:37.261764 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d336ca_3ee7_42d3_855a_a06662e15ebc.slice/crio-c748d506598c74cffb1b2f92db825dc8b1009dd84c3018c41acec018c11d38f1 WatchSource:0}: Error finding container c748d506598c74cffb1b2f92db825dc8b1009dd84c3018c41acec018c11d38f1: Status 404 returned error can't find the container with id c748d506598c74cffb1b2f92db825dc8b1009dd84c3018c41acec018c11d38f1 Oct 30 18:27:37.386239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.386198 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" event={"ID":"ac44c39c-548c-4ab7-a6f2-8f726b047d38","Type":"ContainerStarted","Data":"ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3"} Oct 30 18:27:37.386427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.386252 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" event={"ID":"ac44c39c-548c-4ab7-a6f2-8f726b047d38","Type":"ContainerStarted","Data":"cddbcb6f87e60f74e207666a0abd6c38f6975584ad7632fb2e665253bd0fa438"} Oct 30 18:27:37.386427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.386330 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:37.387803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.387782 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" event={"ID":"32d336ca-3ee7-42d3-855a-a06662e15ebc","Type":"ContainerStarted","Data":"c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a"} Oct 30 18:27:37.387950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.387816 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" event={"ID":"32d336ca-3ee7-42d3-855a-a06662e15ebc","Type":"ContainerStarted","Data":"c748d506598c74cffb1b2f92db825dc8b1009dd84c3018c41acec018c11d38f1"} Oct 30 18:27:37.387950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.387878 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:37.422977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.422931 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" podStartSLOduration=1.422917213 podStartE2EDuration="1.422917213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:37.419920474 +0000 UTC m=+1754.244314766" watchObservedRunningTime="2026-10-30 18:27:37.422917213 +0000 UTC m=+1754.247311545" Oct 30 18:27:37.452725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.452647 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" podStartSLOduration=1.452634208 podStartE2EDuration="1.452634208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:37.451353018 +0000 UTC m=+1754.275747321" watchObservedRunningTime="2026-10-30 18:27:37.452634208 +0000 UTC m=+1754.277028513" Oct 30 18:27:37.554530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:37.554495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:38.695148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:38.695112 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:38.809686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:38.809602 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:39.010518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.010432 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj"] Oct 30 18:27:39.010725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.010697 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" containerID="cri-o://6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87" gracePeriod=30 Oct 30 18:27:39.037373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.037352 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:39.521069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.521037 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:39.630916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.630882 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf"] Oct 30 18:27:39.634151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.634130 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.647034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.647014 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:27:39.649449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.649429 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:39.653174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.653149 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf"] Oct 30 18:27:39.749492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.749456 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.749969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.749669 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9bx\" (UniqueName: \"kubernetes.io/projected/41302665-25ac-4d25-9877-a296ca6cb17b-kube-api-access-wr9bx\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.749969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.749845 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.850719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.850642 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.850892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.850767 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9bx\" (UniqueName: \"kubernetes.io/projected/41302665-25ac-4d25-9877-a296ca6cb17b-kube-api-access-wr9bx\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.850966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.850948 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.851414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.851395 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.853309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.853289 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.866950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.866931 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9bx\" (UniqueName: \"kubernetes.io/projected/41302665-25ac-4d25-9877-a296ca6cb17b-kube-api-access-wr9bx\") pod \"maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:39.948045 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:39.948012 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:40.002032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.002003 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:40.102004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.101910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:40.250571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.250540 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf"] Oct 30 18:27:40.254229 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:40.254200 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41302665_25ac_4d25_9877_a296ca6cb17b.slice/crio-76cedf45e2eb3c8fb7b92313e03b2210a9d8f50173dcb4843cb23dd092493ef2 WatchSource:0}: Error finding container 76cedf45e2eb3c8fb7b92313e03b2210a9d8f50173dcb4843cb23dd092493ef2: Status 404 returned error can't find the container with id 76cedf45e2eb3c8fb7b92313e03b2210a9d8f50173dcb4843cb23dd092493ef2 Oct 30 18:27:40.310730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.310703 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp"] Oct 30 18:27:40.310945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.310921 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" containerID="cri-o://96ccb13fb4e79ce42135b907ed1811660583cc2d61785f0e329a505a3cc632d1" gracePeriod=30 Oct 30 18:27:40.339175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.339148 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:40.408025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.407996 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" event={"ID":"41302665-25ac-4d25-9877-a296ca6cb17b","Type":"ContainerStarted","Data":"76cedf45e2eb3c8fb7b92313e03b2210a9d8f50173dcb4843cb23dd092493ef2"} Oct 30 18:27:40.524720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.524684 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:40.620584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:40.620555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:41.415750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:41.415715 4582 generic.go:356] "Generic (PLEG): container finished" podID="41302665-25ac-4d25-9877-a296ca6cb17b" containerID="0f4f8a50912b5041532001881d19d84286affed3a5ed2e69edb7c7e0a16a47a2" exitCode=0 Oct 30 18:27:41.416224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:41.415778 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" event={"ID":"41302665-25ac-4d25-9877-a296ca6cb17b","Type":"ContainerDied","Data":"0f4f8a50912b5041532001881d19d84286affed3a5ed2e69edb7c7e0a16a47a2"} Oct 30 18:27:42.156767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.156731 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:42.620957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.620931 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:42.692377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.692298 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41302665-25ac-4d25-9877-a296ca6cb17b-kube-api-access-wr9bx\" (UniqueName: \"kubernetes.io/projected/41302665-25ac-4d25-9877-a296ca6cb17b-kube-api-access-wr9bx\") pod \"41302665-25ac-4d25-9877-a296ca6cb17b\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " Oct 30 18:27:42.692543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.692438 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-openshift-ca\") pod \"41302665-25ac-4d25-9877-a296ca6cb17b\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " Oct 30 18:27:42.692617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.692591 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-xks-ca\") pod \"41302665-25ac-4d25-9877-a296ca6cb17b\" (UID: \"41302665-25ac-4d25-9877-a296ca6cb17b\") " Oct 30 18:27:42.692838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.692785 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-openshift-ca" pod "41302665-25ac-4d25-9877-a296ca6cb17b" (UID: "41302665-25ac-4d25-9877-a296ca6cb17b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:42.693118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.693093 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.694932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.694905 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41302665-25ac-4d25-9877-a296ca6cb17b-kube-api-access-wr9bx" pod "41302665-25ac-4d25-9877-a296ca6cb17b" (UID: "41302665-25ac-4d25-9877-a296ca6cb17b"). InnerVolumeSpecName "kube-api-access-wr9bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:42.694932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.694919 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-xks-ca" pod "41302665-25ac-4d25-9877-a296ca6cb17b" (UID: "41302665-25ac-4d25-9877-a296ca6cb17b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:42.794438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.794407 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41302665-25ac-4d25-9877-a296ca6cb17b-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.794438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:42.794437 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9bx\" (UniqueName: \"kubernetes.io/projected/41302665-25ac-4d25-9877-a296ca6cb17b-kube-api-access-wr9bx\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:43.435020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:43.434984 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" event={"ID":"41302665-25ac-4d25-9877-a296ca6cb17b","Type":"ContainerDied","Data":"76cedf45e2eb3c8fb7b92313e03b2210a9d8f50173dcb4843cb23dd092493ef2"} Oct 30 18:27:43.435020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:43.435010 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf" Oct 30 18:27:43.435223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:43.435043 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76cedf45e2eb3c8fb7b92313e03b2210a9d8f50173dcb4843cb23dd092493ef2" Oct 30 18:27:43.950361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:43.950329 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:44.112378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.112340 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:44.332570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.332543 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:27:44.417087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417056 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-podinfo\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417105 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-kube-api-access-v5vrc\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-kube-api-access-v5vrc\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417156 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-data\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417219 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-envoy\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417242 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-certs\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417290 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-token\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-token\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417365 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istiod-ca-cert\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417404 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-credential-socket\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417436 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-socket\") pod \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\" (UID: \"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce\") " Oct 30 18:27:44.417591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417572 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-data" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:44.417809 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417586 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-certs" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:44.417913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417816 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-credential-socket" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:44.417913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.417882 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istiod-ca-cert" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:44.418046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.418025 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-socket" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:44.418259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.418237 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.418259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.418259 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.418414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.418272 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.418414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.418284 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.418414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.418296 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.419943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.419912 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-podinfo" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:44.420063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.420034 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-kube-api-access-v5vrc" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "kube-api-access-v5vrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:44.420177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.420162 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-token" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:44.420319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.420305 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-envoy" pod "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" (UID: "46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:44.467219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.467125 4582 generic.go:356] "Generic (PLEG): container finished" podID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerID="6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87" exitCode=0 Oct 30 18:27:44.467219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.467187 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" Oct 30 18:27:44.467406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.467214 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" event={"ID":"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce","Type":"ContainerDied","Data":"6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87"} Oct 30 18:27:44.467406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.467277 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj" event={"ID":"46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce","Type":"ContainerDied","Data":"be296c478f58ef719223e87ca59db77d0cf8bfe65b67cf0581a8e6d3cc728b6a"} Oct 30 18:27:44.467406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.467309 4582 scope.go:122] "RemoveContainer" containerID="6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87" Oct 30 18:27:44.480063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.479742 4582 scope.go:122] "RemoveContainer" containerID="32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f" Oct 30 18:27:44.490757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.490725 4582 scope.go:122] "RemoveContainer" containerID="6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87" Oct 30 18:27:44.491074 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:44.491054 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87\": container with ID starting with 6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87 not found: ID does not exist" containerID="6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87" Oct 30 18:27:44.491129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.491082 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87"} err="failed to get container status \"6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87\": rpc error: code = NotFound desc = could not find container \"6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87\": container with ID starting with 6d5b4bb39a11610288407bfba584c55502c150d9bcb36bb5770c71fecb2c6c87 not found: ID does not exist" Oct 30 18:27:44.491129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.491100 4582 scope.go:122] "RemoveContainer" containerID="32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f" Oct 30 18:27:44.491323 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:44.491300 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f\": container with ID starting with 32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f not found: ID does not exist" containerID="32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f" Oct 30 18:27:44.491364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.491330 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f"} err="failed to get container status \"32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f\": rpc error: code = NotFound desc = could not find container \"32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f\": container with ID starting with 32404a2d87698433a347324812083361ffe9f3730e35d77f7929c6d5e7d25c5f not found: ID does not exist" Oct 30 18:27:44.512422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.512395 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj"] Oct 30 18:27:44.515971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.515947 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-50b278-openshift-default-6497867cfc-d9nkj"] Oct 30 18:27:44.519550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.519526 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5vrc\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-kube-api-access-v5vrc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.519550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.519550 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.519736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.519560 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:44.519736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:44.519569 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.301435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.301402 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" path="/var/lib/kubelet/pods/46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce/volumes" Oct 30 18:27:45.463844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.463294 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:45.474281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.474248 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:27:45.474453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.474329 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:27:45.478747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.478554 4582 generic.go:356] "Generic (PLEG): container finished" podID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerID="96ccb13fb4e79ce42135b907ed1811660583cc2d61785f0e329a505a3cc632d1" exitCode=0 Oct 30 18:27:45.478747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.478625 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" event={"ID":"3f8a245c-4bcd-410b-be05-47e8e1abd4fb","Type":"ContainerDied","Data":"96ccb13fb4e79ce42135b907ed1811660583cc2d61785f0e329a505a3cc632d1"} Oct 30 18:27:45.478747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.478686 4582 scope.go:122] "RemoveContainer" containerID="f9d2738ae8c5fec7c14743a8a093e431b2977e245644370fd270bb2c238f483f" Oct 30 18:27:45.568577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.568434 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:45.628341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.628319 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:27:45.735310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735271 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-kube-api-access-gn2mv\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-kube-api-access-gn2mv\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735332 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-data\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735548 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-certs\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735586 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-data" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:45.735627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735617 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istiod-ca-cert\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735652 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-envoy\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735680 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-socket\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735762 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-token\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-token\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735843 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-credential-socket\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.735890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735839 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-certs" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:45.735890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735887 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-podinfo\") pod \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\" (UID: \"3f8a245c-4bcd-410b-be05-47e8e1abd4fb\") " Oct 30 18:27:45.736126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735961 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-socket" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:45.736126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.735995 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istiod-ca-cert" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:45.736226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.736126 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-credential-socket" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:45.736520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.736496 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.736520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.736524 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.736727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.736537 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.736727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.736550 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.736727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.736564 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.738214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.738185 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-envoy" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:45.738343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.738247 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-podinfo" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:45.738526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.738508 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-kube-api-access-gn2mv" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "kube-api-access-gn2mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:45.738584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.738528 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-token" pod "3f8a245c-4bcd-410b-be05-47e8e1abd4fb" (UID: "3f8a245c-4bcd-410b-be05-47e8e1abd4fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:45.837462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.837386 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.837462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.837411 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.837462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.837423 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2mv\" (UniqueName: \"kubernetes.io/projected/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-kube-api-access-gn2mv\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.837462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.837437 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8a245c-4bcd-410b-be05-47e8e1abd4fb-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:27:45.859854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:45.854665 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:46.487148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:46.487118 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" Oct 30 18:27:46.487597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:46.487147 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp" event={"ID":"3f8a245c-4bcd-410b-be05-47e8e1abd4fb","Type":"ContainerDied","Data":"cd9eea4a41ecd7b2e6b11d2988188998d83d7ad2ac6d55a023a4bf15029b03e0"} Oct 30 18:27:46.487597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:46.487201 4582 scope.go:122] "RemoveContainer" containerID="96ccb13fb4e79ce42135b907ed1811660583cc2d61785f0e329a505a3cc632d1" Oct 30 18:27:46.540262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:46.540228 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp"] Oct 30 18:27:46.569182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:46.569145 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f42594-openshift-default-c8d4dd6d7-ffjhp"] Oct 30 18:27:47.300241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:47.300209 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" path="/var/lib/kubelet/pods/3f8a245c-4bcd-410b-be05-47e8e1abd4fb/volumes" Oct 30 18:27:47.581363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:47.581275 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:48.398220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:48.398190 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:27:48.398440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:48.398299 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:27:49.429729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:49.429694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:50.026941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:50.026907 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:50.124068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:50.123962 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:51.766202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:51.766164 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.527042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:52.526965 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.640801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:52.640762 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.680898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:52.680765 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf"] Oct 30 18:27:52.689258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:52.689230 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-afc614-qh5zf"] Oct 30 18:27:52.723650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:52.723566 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.130634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.130610 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.300351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.300285 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41302665-25ac-4d25-9877-a296ca6cb17b" path="/var/lib/kubelet/pods/41302665-25ac-4d25-9877-a296ca6cb17b/volumes" Oct 30 18:27:53.574538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.574504 4582 generic.go:356] "Generic (PLEG): container finished" podID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerID="4ef30f742dcc2cd71f2c09724d0154063f9542b7730e02c11588f9d357efb16e" exitCode=0 Oct 30 18:27:53.574722 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.574574 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" event={"ID":"9defb28d-0725-409e-80d5-ba3b0df5d4fd","Type":"ContainerDied","Data":"4ef30f742dcc2cd71f2c09724d0154063f9542b7730e02c11588f9d357efb16e"} Oct 30 18:27:53.575069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.575053 4582 scope.go:122] "RemoveContainer" containerID="4ef30f742dcc2cd71f2c09724d0154063f9542b7730e02c11588f9d357efb16e" Oct 30 18:27:53.576667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.576642 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:27:53.576732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.576716 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:27:53.576769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:53.576754 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:27:54.433572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:54.433519 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:54.533255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:54.533225 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:54.586532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:54.586500 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" event={"ID":"9defb28d-0725-409e-80d5-ba3b0df5d4fd","Type":"ContainerStarted","Data":"74d154ff598ef652f42446ce055b19d8a513dfd8cb0818b5ca0b36a4685f544b"} Oct 30 18:27:55.021905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:55.021864 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:27:55.023280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:55.023258 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:27:55.023362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:55.023307 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 18:27:56.022449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.022416 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:27:56.022851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.022469 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 18:27:56.261845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.261165 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:56.394011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.393977 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds"] Oct 30 18:27:56.394561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394548 4582 cpu_manager.go:414] "containerMap: removing container" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" Oct 30 18:27:56.394620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394562 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" Oct 30 18:27:56.394620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394582 4582 cpu_manager.go:414] "containerMap: removing container" podUID="41302665-25ac-4d25-9877-a296ca6cb17b" containerName="revoke-keys" Oct 30 18:27:56.394620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394596 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41302665-25ac-4d25-9877-a296ca6cb17b" containerName="revoke-keys" Oct 30 18:27:56.394620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394612 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" Oct 30 18:27:56.394620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394617 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394635 4582 cpu_manager.go:414] "containerMap: removing container" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394640 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394651 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394656 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394734 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394746 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394752 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8a245c-4bcd-410b-be05-47e8e1abd4fb" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394760 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fbc805-f2bd-4dcb-98b7-3d5994e3b9ce" containerName="istio-proxy" Oct 30 18:27:56.394783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.394769 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="41302665-25ac-4d25-9877-a296ca6cb17b" containerName="revoke-keys" Oct 30 18:27:56.397436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.397416 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.421736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.421710 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:56.421857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.421753 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:27:56.425213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.425192 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds"] Oct 30 18:27:56.514896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.514866 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.515074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.515002 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46dp8\" (UniqueName: \"kubernetes.io/projected/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-kube-api-access-46dp8\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.515128 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.515087 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.616285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.616244 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.616487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.616405 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46dp8\" (UniqueName: \"kubernetes.io/projected/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-kube-api-access-46dp8\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.616487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.616463 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.616933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.616915 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.618825 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.618796 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.632242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.632215 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46dp8\" (UniqueName: \"kubernetes.io/projected/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-kube-api-access-46dp8\") pod \"maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:56.710543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:56.710465 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:57.023080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.023005 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:27:57.023080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.023058 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 18:27:57.027998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.027910 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds"] Oct 30 18:27:57.031018 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:57.030985 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b23ff8a_0060_42d7_8b5d_e0d24b2abc70.slice/crio-a03d98e846f3b28e9bbff213acb7660934ad98d78d79b8dccf19a4e41f45bd79 WatchSource:0}: Error finding container a03d98e846f3b28e9bbff213acb7660934ad98d78d79b8dccf19a4e41f45bd79: Status 404 returned error can't find the container with id a03d98e846f3b28e9bbff213acb7660934ad98d78d79b8dccf19a4e41f45bd79 Oct 30 18:27:57.061865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.058473 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:27:57.608069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.607986 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" event={"ID":"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70","Type":"ContainerStarted","Data":"24725b85f00d939264278248afb548651d91ad4a855b9f2067233c2a5b2ccd86"} Oct 30 18:27:57.608069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.608041 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" event={"ID":"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70","Type":"ContainerStarted","Data":"a03d98e846f3b28e9bbff213acb7660934ad98d78d79b8dccf19a4e41f45bd79"} Oct 30 18:27:57.668740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.667710 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" podStartSLOduration=1.667691327 podStartE2EDuration="1.667691327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:57.640233398 +0000 UTC m=+1774.464627690" watchObservedRunningTime="2026-10-30 18:27:57.667691327 +0000 UTC m=+1774.492085670" Oct 30 18:27:57.935390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.935355 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j"] Oct 30 18:27:57.938562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.938538 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:57.960844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.958521 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-44379a-serving-cert\"" Oct 30 18:27:57.970197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:57.970108 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j"] Oct 30 18:27:58.014564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.014520 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:58.021521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.021489 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:27:58.021646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.021549 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 18:27:58.033484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.033459 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.033834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.033643 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.033834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.033750 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.033834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.033799 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lwm\" (UniqueName: \"kubernetes.io/projected/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-kube-api-access-v4lwm\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.108522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.108490 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:58.134721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.134685 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.134890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.134795 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.134890 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:58.134836 4582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-serving-cert: secret "maas-api-e2e-ipp-b-44379a-serving-cert" not found Oct 30 18:27:58.134890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.134868 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lwm\" (UniqueName: \"kubernetes.io/projected/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-kube-api-access-v4lwm\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.135067 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:27:58.134906 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls podName:e941aa1e-9529-4b0c-b7b2-e70e63bb53b2 nodeName:}" failed. No retries permitted until 2026-10-30 18:27:58.634885559 +0000 UTC m=+1775.459279834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls") pod "maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" (UID: "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2") : secret "maas-api-e2e-ipp-b-44379a-serving-cert" not found Oct 30 18:27:58.135067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.135025 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.135938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.135894 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.137864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.137809 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.167708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.167643 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lwm\" (UniqueName: \"kubernetes.io/projected/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-kube-api-access-v4lwm\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.302113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.302076 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:58.620481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.620443 4582 generic.go:356] "Generic (PLEG): container finished" podID="5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" containerID="24725b85f00d939264278248afb548651d91ad4a855b9f2067233c2a5b2ccd86" exitCode=0 Oct 30 18:27:58.620651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.620530 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" event={"ID":"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70","Type":"ContainerDied","Data":"24725b85f00d939264278248afb548651d91ad4a855b9f2067233c2a5b2ccd86"} Oct 30 18:27:58.641712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.641685 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.671299 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.671263 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:58.883905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:58.883788 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:27:59.021519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.021487 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:27:59.021701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.021549 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 18:27:59.201559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.201471 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j"] Oct 30 18:27:59.210483 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:27:59.210451 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode941aa1e_9529_4b0c_b7b2_e70e63bb53b2.slice/crio-e202f2be8de4063f67b38e1e0d0523e1bf3871662441bf3bfa3c176b661663d1 WatchSource:0}: Error finding container e202f2be8de4063f67b38e1e0d0523e1bf3871662441bf3bfa3c176b661663d1: Status 404 returned error can't find the container with id e202f2be8de4063f67b38e1e0d0523e1bf3871662441bf3bfa3c176b661663d1 Oct 30 18:27:59.415865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.415836 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:59.513190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.513102 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:59.629869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.629831 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" event={"ID":"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2","Type":"ContainerStarted","Data":"e202f2be8de4063f67b38e1e0d0523e1bf3871662441bf3bfa3c176b661663d1"} Oct 30 18:27:59.732650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.732609 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4"] Oct 30 18:27:59.739313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.738230 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.758374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.758194 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4"] Oct 30 18:27:59.779884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.779789 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:59.785213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.785189 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.785362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.785250 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.785484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.785464 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbqv\" (UniqueName: \"kubernetes.io/projected/133eb005-69a1-4688-bff4-1cc0792e894a-kube-api-access-5xbqv\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.834976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.834954 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:27:59.896872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.895600 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.896872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.895769 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbqv\" (UniqueName: \"kubernetes.io/projected/133eb005-69a1-4688-bff4-1cc0792e894a-kube-api-access-5xbqv\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.896872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.895906 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.896872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.896740 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.946376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.946324 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.946932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.946907 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbqv\" (UniqueName: \"kubernetes.io/projected/133eb005-69a1-4688-bff4-1cc0792e894a-kube-api-access-5xbqv\") pod \"maas-api-revoke-sub-a41300a1-9wgt4\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:27:59.996768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.996740 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-xks-ca\") pod \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " Oct 30 18:27:59.996969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.996952 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-kube-api-access-46dp8\" (UniqueName: \"kubernetes.io/projected/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-kube-api-access-46dp8\") pod \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " Oct 30 18:27:59.997043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.996998 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-openshift-ca\") pod \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\" (UID: \"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70\") " Oct 30 18:27:59.997402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.997374 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-openshift-ca" pod "5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" (UID: "5b23ff8a-0060-42d7-8b5d-e0d24b2abc70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:59.998729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.998704 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-xks-ca" pod "5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" (UID: "5b23ff8a-0060-42d7-8b5d-e0d24b2abc70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:59.999124 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:27:59.999106 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-kube-api-access-46dp8" pod "5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" (UID: "5b23ff8a-0060-42d7-8b5d-e0d24b2abc70"). InnerVolumeSpecName "kube-api-access-46dp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:00.027098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.027075 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:28:00.028264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.028243 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:28:00.080958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.080881 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:28:00.098452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.098420 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:00.098452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.098444 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46dp8\" (UniqueName: \"kubernetes.io/projected/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-kube-api-access-46dp8\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:00.098647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.098457 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:00.108880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.107935 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:00.388723 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.388516 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4"] Oct 30 18:28:00.391834 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:00.391793 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133eb005_69a1_4688_bff4_1cc0792e894a.slice/crio-48e03da665cf7de5ff073c37a145f6fe3082a679060a26ec7285193d8048864d WatchSource:0}: Error finding container 48e03da665cf7de5ff073c37a145f6fe3082a679060a26ec7285193d8048864d: Status 404 returned error can't find the container with id 48e03da665cf7de5ff073c37a145f6fe3082a679060a26ec7285193d8048864d Oct 30 18:28:00.637647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.637612 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" event={"ID":"133eb005-69a1-4688-bff4-1cc0792e894a","Type":"ContainerStarted","Data":"48e03da665cf7de5ff073c37a145f6fe3082a679060a26ec7285193d8048864d"} Oct 30 18:28:00.639030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.638953 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" event={"ID":"5b23ff8a-0060-42d7-8b5d-e0d24b2abc70","Type":"ContainerDied","Data":"a03d98e846f3b28e9bbff213acb7660934ad98d78d79b8dccf19a4e41f45bd79"} Oct 30 18:28:00.639030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.638983 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds" Oct 30 18:28:00.639030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:00.638997 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a03d98e846f3b28e9bbff213acb7660934ad98d78d79b8dccf19a4e41f45bd79" Oct 30 18:28:01.647103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:01.647066 4582 generic.go:356] "Generic (PLEG): container finished" podID="133eb005-69a1-4688-bff4-1cc0792e894a" containerID="08d4b9891ff2b5603b241f3fe4b431e77dad1d9602865e8dd98ab5fb716bb795" exitCode=0 Oct 30 18:28:01.647556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:01.647154 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" event={"ID":"133eb005-69a1-4688-bff4-1cc0792e894a","Type":"ContainerDied","Data":"08d4b9891ff2b5603b241f3fe4b431e77dad1d9602865e8dd98ab5fb716bb795"} Oct 30 18:28:01.648877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:01.648850 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" event={"ID":"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2","Type":"ContainerStarted","Data":"991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f"} Oct 30 18:28:01.649006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:01.648974 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:28:01.714244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:01.714201 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" podStartSLOduration=3.146517676 podStartE2EDuration="4.714189597s" totalImagesPullingTime="1.567671921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:01.710945188 +0000 UTC m=+1778.535339479" watchObservedRunningTime="2026-10-30 18:28:01.714189597 +0000 UTC m=+1778.538583889" Oct 30 18:28:01.854038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:01.853997 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:02.803213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.803187 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:28:02.833612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.833581 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133eb005-69a1-4688-bff4-1cc0792e894a-kube-api-access-5xbqv\" (UniqueName: \"kubernetes.io/projected/133eb005-69a1-4688-bff4-1cc0792e894a-kube-api-access-5xbqv\") pod \"133eb005-69a1-4688-bff4-1cc0792e894a\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " Oct 30 18:28:02.833742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.833690 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-xks-ca\") pod \"133eb005-69a1-4688-bff4-1cc0792e894a\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " Oct 30 18:28:02.833836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.833777 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-openshift-ca\") pod \"133eb005-69a1-4688-bff4-1cc0792e894a\" (UID: \"133eb005-69a1-4688-bff4-1cc0792e894a\") " Oct 30 18:28:02.834161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.834134 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-openshift-ca" pod "133eb005-69a1-4688-bff4-1cc0792e894a" (UID: "133eb005-69a1-4688-bff4-1cc0792e894a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:02.834362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.834327 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:02.836053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.836030 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133eb005-69a1-4688-bff4-1cc0792e894a-kube-api-access-5xbqv" pod "133eb005-69a1-4688-bff4-1cc0792e894a" (UID: "133eb005-69a1-4688-bff4-1cc0792e894a"). InnerVolumeSpecName "kube-api-access-5xbqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:02.836221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.836199 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-xks-ca" pod "133eb005-69a1-4688-bff4-1cc0792e894a" (UID: "133eb005-69a1-4688-bff4-1cc0792e894a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:02.936040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.935959 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xbqv\" (UniqueName: \"kubernetes.io/projected/133eb005-69a1-4688-bff4-1cc0792e894a-kube-api-access-5xbqv\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:02.936040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:02.935987 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133eb005-69a1-4688-bff4-1cc0792e894a-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:03.611342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:03.611247 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:03.663882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:03.663849 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" event={"ID":"133eb005-69a1-4688-bff4-1cc0792e894a","Type":"ContainerDied","Data":"48e03da665cf7de5ff073c37a145f6fe3082a679060a26ec7285193d8048864d"} Oct 30 18:28:03.663882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:03.663871 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4" Oct 30 18:28:03.664091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:03.663903 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e03da665cf7de5ff073c37a145f6fe3082a679060a26ec7285193d8048864d" Oct 30 18:28:04.019246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.019205 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5"] Oct 30 18:28:04.019842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.019815 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" containerName="revoke-keys" Oct 30 18:28:04.019900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.019843 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" containerName="revoke-keys" Oct 30 18:28:04.019900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.019875 4582 cpu_manager.go:414] "containerMap: removing container" podUID="133eb005-69a1-4688-bff4-1cc0792e894a" containerName="revoke-keys" Oct 30 18:28:04.019900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.019881 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133eb005-69a1-4688-bff4-1cc0792e894a" containerName="revoke-keys" Oct 30 18:28:04.020014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.019989 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="133eb005-69a1-4688-bff4-1cc0792e894a" containerName="revoke-keys" Oct 30 18:28:04.020014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.020006 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" containerName="revoke-keys" Oct 30 18:28:04.023562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.023545 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.037397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.037377 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:04.041070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.041047 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5"] Oct 30 18:28:04.046561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.046540 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:28:04.049087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.049049 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.049244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.049223 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgqt\" (UniqueName: \"kubernetes.io/projected/7b2d9f8a-3f69-402b-b329-303d4a7419fc-kube-api-access-4qgqt\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.049386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.049370 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.150935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.150903 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgqt\" (UniqueName: \"kubernetes.io/projected/7b2d9f8a-3f69-402b-b329-303d4a7419fc-kube-api-access-4qgqt\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.151127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.150991 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.151127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.151052 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.151595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.151573 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.153412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.153386 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.167217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.167195 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgqt\" (UniqueName: \"kubernetes.io/projected/7b2d9f8a-3f69-402b-b329-303d4a7419fc-kube-api-access-4qgqt\") pod \"maas-api-revoke-sub-5ea59957-rmnq5\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.338039 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.337967 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:04.638240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.638071 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5"] Oct 30 18:28:04.641363 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:04.641337 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2d9f8a_3f69_402b_b329_303d4a7419fc.slice/crio-51cd24898ce8527539b95603921b9e6319987bd12f8fe2e5a83c924a5f55d24a WatchSource:0}: Error finding container 51cd24898ce8527539b95603921b9e6319987bd12f8fe2e5a83c924a5f55d24a: Status 404 returned error can't find the container with id 51cd24898ce8527539b95603921b9e6319987bd12f8fe2e5a83c924a5f55d24a Oct 30 18:28:04.671903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:04.671866 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" event={"ID":"7b2d9f8a-3f69-402b-b329-303d4a7419fc","Type":"ContainerStarted","Data":"51cd24898ce8527539b95603921b9e6319987bd12f8fe2e5a83c924a5f55d24a"} Oct 30 18:28:05.679933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:05.679890 4582 generic.go:356] "Generic (PLEG): container finished" podID="7b2d9f8a-3f69-402b-b329-303d4a7419fc" containerID="3c559a7a6824ee0fc295c1bb3f253ad083b9d53c688a54fc01330f7a929c4eeb" exitCode=0 Oct 30 18:28:05.680330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:05.680038 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" event={"ID":"7b2d9f8a-3f69-402b-b329-303d4a7419fc","Type":"ContainerDied","Data":"3c559a7a6824ee0fc295c1bb3f253ad083b9d53c688a54fc01330f7a929c4eeb"} Oct 30 18:28:05.891426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:05.891392 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:06.225246 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:06.225157 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:28:06.857707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:06.857676 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:06.983203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:06.983108 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-xks-ca\") pod \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " Oct 30 18:28:06.983203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:06.983194 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2d9f8a-3f69-402b-b329-303d4a7419fc-kube-api-access-4qgqt\" (UniqueName: \"kubernetes.io/projected/7b2d9f8a-3f69-402b-b329-303d4a7419fc-kube-api-access-4qgqt\") pod \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " Oct 30 18:28:06.983489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:06.983470 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-openshift-ca\") pod \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\" (UID: \"7b2d9f8a-3f69-402b-b329-303d4a7419fc\") " Oct 30 18:28:06.983965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:06.983938 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-openshift-ca" pod "7b2d9f8a-3f69-402b-b329-303d4a7419fc" (UID: "7b2d9f8a-3f69-402b-b329-303d4a7419fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:06.986225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:06.986197 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-xks-ca" pod "7b2d9f8a-3f69-402b-b329-303d4a7419fc" (UID: "7b2d9f8a-3f69-402b-b329-303d4a7419fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:06.986399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:06.986282 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2d9f8a-3f69-402b-b329-303d4a7419fc-kube-api-access-4qgqt" pod "7b2d9f8a-3f69-402b-b329-303d4a7419fc" (UID: "7b2d9f8a-3f69-402b-b329-303d4a7419fc"). InnerVolumeSpecName "kube-api-access-4qgqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:07.017858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.017797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:07.085020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.084987 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:07.085175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.085022 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2d9f8a-3f69-402b-b329-303d4a7419fc-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:07.085175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.085048 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qgqt\" (UniqueName: \"kubernetes.io/projected/7b2d9f8a-3f69-402b-b329-303d4a7419fc-kube-api-access-4qgqt\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:07.123070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.123028 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:07.661885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.661853 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:28:07.691900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.691865 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:07.696711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.696684 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" Oct 30 18:28:07.696711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.696703 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5" event={"ID":"7b2d9f8a-3f69-402b-b329-303d4a7419fc","Type":"ContainerDied","Data":"51cd24898ce8527539b95603921b9e6319987bd12f8fe2e5a83c924a5f55d24a"} Oct 30 18:28:07.696986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:07.696756 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51cd24898ce8527539b95603921b9e6319987bd12f8fe2e5a83c924a5f55d24a" Oct 30 18:28:09.467810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:09.467778 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:10.097276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:10.097235 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:11.275763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:11.275725 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:13.015632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:13.015579 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:14.791648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:14.791614 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:15.073483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.073392 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm"] Oct 30 18:28:15.073670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.073596 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" podUID="09911906-7771-4ad2-b09d-6d0d3f6f5008" containerName="maas-api" containerID="cri-o://5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a" gracePeriod=30 Oct 30 18:28:15.124439 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.124405 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:15.374603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.374579 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:28:15.484326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.484291 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-metrics-certs\") pod \"09911906-7771-4ad2-b09d-6d0d3f6f5008\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " Oct 30 18:28:15.484525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.484356 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09911906-7771-4ad2-b09d-6d0d3f6f5008-kube-api-access-g7hfw\" (UniqueName: \"kubernetes.io/projected/09911906-7771-4ad2-b09d-6d0d3f6f5008-kube-api-access-g7hfw\") pod \"09911906-7771-4ad2-b09d-6d0d3f6f5008\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " Oct 30 18:28:15.484525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.484418 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-maas-api-tls\") pod \"09911906-7771-4ad2-b09d-6d0d3f6f5008\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " Oct 30 18:28:15.484525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.484481 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09911906-7771-4ad2-b09d-6d0d3f6f5008-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09911906-7771-4ad2-b09d-6d0d3f6f5008-openshift-service-ca\") pod \"09911906-7771-4ad2-b09d-6d0d3f6f5008\" (UID: \"09911906-7771-4ad2-b09d-6d0d3f6f5008\") " Oct 30 18:28:15.484928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.484899 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09911906-7771-4ad2-b09d-6d0d3f6f5008-openshift-service-ca" pod "09911906-7771-4ad2-b09d-6d0d3f6f5008" (UID: "09911906-7771-4ad2-b09d-6d0d3f6f5008"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:15.485098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.485078 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09911906-7771-4ad2-b09d-6d0d3f6f5008-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:15.486513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.486488 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-metrics-certs" pod "09911906-7771-4ad2-b09d-6d0d3f6f5008" (UID: "09911906-7771-4ad2-b09d-6d0d3f6f5008"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:15.486606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.486510 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09911906-7771-4ad2-b09d-6d0d3f6f5008-kube-api-access-g7hfw" pod "09911906-7771-4ad2-b09d-6d0d3f6f5008" (UID: "09911906-7771-4ad2-b09d-6d0d3f6f5008"). InnerVolumeSpecName "kube-api-access-g7hfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:15.486606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.486522 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-maas-api-tls" pod "09911906-7771-4ad2-b09d-6d0d3f6f5008" (UID: "09911906-7771-4ad2-b09d-6d0d3f6f5008"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:15.586393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.586361 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:15.586393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.586388 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7hfw\" (UniqueName: \"kubernetes.io/projected/09911906-7771-4ad2-b09d-6d0d3f6f5008-kube-api-access-g7hfw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:15.586393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.586398 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09911906-7771-4ad2-b09d-6d0d3f6f5008-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:15.747273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.747234 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:15.753988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.753954 4582 generic.go:356] "Generic (PLEG): container finished" podID="09911906-7771-4ad2-b09d-6d0d3f6f5008" containerID="5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a" exitCode=0 Oct 30 18:28:15.754149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.753994 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" event={"ID":"09911906-7771-4ad2-b09d-6d0d3f6f5008","Type":"ContainerDied","Data":"5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a"} Oct 30 18:28:15.754149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.754012 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" Oct 30 18:28:15.754149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.754055 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm" event={"ID":"09911906-7771-4ad2-b09d-6d0d3f6f5008","Type":"ContainerDied","Data":"9062269a798c83da9dc931a08929719c929d512707eec182e5823bfec20d9109"} Oct 30 18:28:15.754149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.754088 4582 scope.go:122] "RemoveContainer" containerID="5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a" Oct 30 18:28:15.765224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.765205 4582 scope.go:122] "RemoveContainer" containerID="5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a" Oct 30 18:28:15.765498 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:15.765477 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a\": container with ID starting with 5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a not found: ID does not exist" containerID="5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a" Oct 30 18:28:15.765596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.765508 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a"} err="failed to get container status \"5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a\": rpc error: code = NotFound desc = could not find container \"5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a\": container with ID starting with 5c73d7900340bc1299b6914497f0c127cb940a363544bd0e58c29bb336241c3a not found: ID does not exist" Oct 30 18:28:15.801839 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.801797 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm"] Oct 30 18:28:15.806506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.806484 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be7f9e-995859f69-xbzbm"] Oct 30 18:28:15.850277 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.850230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:15.931990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.931953 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds"] Oct 30 18:28:15.935910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.935883 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-be7f9e-p87ds"] Oct 30 18:28:15.958526 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:15.958493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:16.077760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.077683 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz"] Oct 30 18:28:16.078289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.078274 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7b2d9f8a-3f69-402b-b329-303d4a7419fc" containerName="revoke-keys" Oct 30 18:28:16.078342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.078291 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2d9f8a-3f69-402b-b329-303d4a7419fc" containerName="revoke-keys" Oct 30 18:28:16.078342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.078329 4582 cpu_manager.go:414] "containerMap: removing container" podUID="09911906-7771-4ad2-b09d-6d0d3f6f5008" containerName="maas-api" Oct 30 18:28:16.078342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.078334 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09911906-7771-4ad2-b09d-6d0d3f6f5008" containerName="maas-api" Oct 30 18:28:16.078449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.078423 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="09911906-7771-4ad2-b09d-6d0d3f6f5008" containerName="maas-api" Oct 30 18:28:16.078449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.078436 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2d9f8a-3f69-402b-b329-303d4a7419fc" containerName="revoke-keys" Oct 30 18:28:16.082854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.082814 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.095916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.095895 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:16.095916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.095905 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:28:16.099477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.099454 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz"] Oct 30 18:28:16.192520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.192482 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.192740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.192720 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49brd\" (UniqueName: \"kubernetes.io/projected/f4111b3e-d0fc-42f2-932f-db9ba41043d3-kube-api-access-49brd\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.192785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.192759 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.293510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.293479 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49brd\" (UniqueName: \"kubernetes.io/projected/f4111b3e-d0fc-42f2-932f-db9ba41043d3-kube-api-access-49brd\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.293510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.293515 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.293746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.293553 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.294290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.294261 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.295790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.295772 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.309890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.309868 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49brd\" (UniqueName: \"kubernetes.io/projected/f4111b3e-d0fc-42f2-932f-db9ba41043d3-kube-api-access-49brd\") pod \"maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.396759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.396739 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:16.415288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.415259 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:16.696013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.695935 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz"] Oct 30 18:28:16.699030 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:16.698997 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4111b3e_d0fc_42f2_932f_db9ba41043d3.slice/crio-3cd205401bff7271a3263f605d1482e176ffa218bb650f890b29dcaef05d1886 WatchSource:0}: Error finding container 3cd205401bff7271a3263f605d1482e176ffa218bb650f890b29dcaef05d1886: Status 404 returned error can't find the container with id 3cd205401bff7271a3263f605d1482e176ffa218bb650f890b29dcaef05d1886 Oct 30 18:28:16.771597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:16.771564 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" event={"ID":"f4111b3e-d0fc-42f2-932f-db9ba41043d3","Type":"ContainerStarted","Data":"3cd205401bff7271a3263f605d1482e176ffa218bb650f890b29dcaef05d1886"} Oct 30 18:28:17.305574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:17.305493 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09911906-7771-4ad2-b09d-6d0d3f6f5008" path="/var/lib/kubelet/pods/09911906-7771-4ad2-b09d-6d0d3f6f5008/volumes" Oct 30 18:28:17.306040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:17.306012 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b23ff8a-0060-42d7-8b5d-e0d24b2abc70" path="/var/lib/kubelet/pods/5b23ff8a-0060-42d7-8b5d-e0d24b2abc70/volumes" Oct 30 18:28:17.761811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:17.761756 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:17.781052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:17.781023 4582 generic.go:356] "Generic (PLEG): container finished" podID="f4111b3e-d0fc-42f2-932f-db9ba41043d3" containerID="3a7188bff5eabd8fb41e89b4dedc2d001d41023aeacfe0ab68e05ad5ab077e04" exitCode=0 Oct 30 18:28:17.781201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:17.781105 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" event={"ID":"f4111b3e-d0fc-42f2-932f-db9ba41043d3","Type":"ContainerDied","Data":"3a7188bff5eabd8fb41e89b4dedc2d001d41023aeacfe0ab68e05ad5ab077e04"} Oct 30 18:28:18.953186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:18.953164 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:19.029404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.029371 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-openshift-ca\") pod \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " Oct 30 18:28:19.029570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.029461 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4111b3e-d0fc-42f2-932f-db9ba41043d3-kube-api-access-49brd\" (UniqueName: \"kubernetes.io/projected/f4111b3e-d0fc-42f2-932f-db9ba41043d3-kube-api-access-49brd\") pod \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " Oct 30 18:28:19.029570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.029521 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-xks-ca\") pod \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\" (UID: \"f4111b3e-d0fc-42f2-932f-db9ba41043d3\") " Oct 30 18:28:19.029793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.029771 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-openshift-ca" pod "f4111b3e-d0fc-42f2-932f-db9ba41043d3" (UID: "f4111b3e-d0fc-42f2-932f-db9ba41043d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:19.030185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.030158 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:19.031615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.031590 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-xks-ca" pod "f4111b3e-d0fc-42f2-932f-db9ba41043d3" (UID: "f4111b3e-d0fc-42f2-932f-db9ba41043d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:19.031737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.031721 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4111b3e-d0fc-42f2-932f-db9ba41043d3-kube-api-access-49brd" pod "f4111b3e-d0fc-42f2-932f-db9ba41043d3" (UID: "f4111b3e-d0fc-42f2-932f-db9ba41043d3"). InnerVolumeSpecName "kube-api-access-49brd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:19.131291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.131260 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49brd\" (UniqueName: \"kubernetes.io/projected/f4111b3e-d0fc-42f2-932f-db9ba41043d3-kube-api-access-49brd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:19.131291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.131290 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4111b3e-d0fc-42f2-932f-db9ba41043d3-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:19.556246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.556169 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:19.801501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.801463 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" Oct 30 18:28:19.801501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.801485 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz" event={"ID":"f4111b3e-d0fc-42f2-932f-db9ba41043d3","Type":"ContainerDied","Data":"3cd205401bff7271a3263f605d1482e176ffa218bb650f890b29dcaef05d1886"} Oct 30 18:28:19.801738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:19.801529 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd205401bff7271a3263f605d1482e176ffa218bb650f890b29dcaef05d1886" Oct 30 18:28:21.669481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:21.669436 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:22.490471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:22.490434 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:22.601704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:22.601673 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:22.847476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:22.847400 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:23.510614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.510583 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n"] Oct 30 18:28:23.511180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.511165 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f4111b3e-d0fc-42f2-932f-db9ba41043d3" containerName="revoke-keys" Oct 30 18:28:23.511250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.511180 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4111b3e-d0fc-42f2-932f-db9ba41043d3" containerName="revoke-keys" Oct 30 18:28:23.511310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.511301 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4111b3e-d0fc-42f2-932f-db9ba41043d3" containerName="revoke-keys" Oct 30 18:28:23.515030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.515015 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.534302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.533069 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n"] Oct 30 18:28:23.538846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.538801 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:23.538970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.538951 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:28:23.680027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.679998 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:23.694552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.694522 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tk6f\" (UniqueName: \"kubernetes.io/projected/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-kube-api-access-6tk6f\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.694669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.694569 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.694669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.694636 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.796227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.796152 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tk6f\" (UniqueName: \"kubernetes.io/projected/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-kube-api-access-6tk6f\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.796227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.796202 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.796451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.796231 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.796903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.796876 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.798615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.798597 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.813222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.813198 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tk6f\" (UniqueName: \"kubernetes.io/projected/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-kube-api-access-6tk6f\") pod \"maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:23.828563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:23.828542 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:24.126891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:24.126856 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n"] Oct 30 18:28:24.129643 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:24.129607 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11f8d2b_c019_4fbf_be90_b679fbdbfdcb.slice/crio-8be79fb2d34f698a5004a36e66b78562418ef3c1db1226fb1d970b94f6898c7c WatchSource:0}: Error finding container 8be79fb2d34f698a5004a36e66b78562418ef3c1db1226fb1d970b94f6898c7c: Status 404 returned error can't find the container with id 8be79fb2d34f698a5004a36e66b78562418ef3c1db1226fb1d970b94f6898c7c Oct 30 18:28:24.844076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:24.843990 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" event={"ID":"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb","Type":"ContainerStarted","Data":"248a7afb1d639d955affb1c80df685d47fe07986cb86121eb1ab91892d5edd11"} Oct 30 18:28:24.844076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:24.844037 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" event={"ID":"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb","Type":"ContainerStarted","Data":"8be79fb2d34f698a5004a36e66b78562418ef3c1db1226fb1d970b94f6898c7c"} Oct 30 18:28:24.879839 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:24.879731 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" podStartSLOduration=1.879717598 podStartE2EDuration="1.879717598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:24.877809115 +0000 UTC m=+1801.702203407" watchObservedRunningTime="2026-10-30 18:28:24.879717598 +0000 UTC m=+1801.704111948" Oct 30 18:28:25.152053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.152023 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7"] Oct 30 18:28:25.192711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.192676 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl"] Oct 30 18:28:25.292524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.292486 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.388294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.388261 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.488424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.488344 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.692701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.692665 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.819754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.819675 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.852313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.852276 4582 generic.go:356] "Generic (PLEG): container finished" podID="d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" containerID="248a7afb1d639d955affb1c80df685d47fe07986cb86121eb1ab91892d5edd11" exitCode=0 Oct 30 18:28:25.852512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:25.852318 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" event={"ID":"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb","Type":"ContainerDied","Data":"248a7afb1d639d955affb1c80df685d47fe07986cb86121eb1ab91892d5edd11"} Oct 30 18:28:27.018537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.018514 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:27.032860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.032815 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-openshift-ca\") pod \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " Oct 30 18:28:27.033089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.033067 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-openshift-ca" pod "d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" (UID: "d11f8d2b-c019-4fbf-be90-b679fbdbfdcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:27.033145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.033060 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-kube-api-access-6tk6f\" (UniqueName: \"kubernetes.io/projected/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-kube-api-access-6tk6f\") pod \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " Oct 30 18:28:27.033191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.033173 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-xks-ca\") pod \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\" (UID: \"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb\") " Oct 30 18:28:27.033545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.033528 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:27.035041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.035018 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-kube-api-access-6tk6f" pod "d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" (UID: "d11f8d2b-c019-4fbf-be90-b679fbdbfdcb"). InnerVolumeSpecName "kube-api-access-6tk6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:27.035144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.035094 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-xks-ca" pod "d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" (UID: "d11f8d2b-c019-4fbf-be90-b679fbdbfdcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:27.134894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.134865 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tk6f\" (UniqueName: \"kubernetes.io/projected/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-kube-api-access-6tk6f\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:27.134894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.134892 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:27.867029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.866996 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" Oct 30 18:28:27.867196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.867034 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n" event={"ID":"d11f8d2b-c019-4fbf-be90-b679fbdbfdcb","Type":"ContainerDied","Data":"8be79fb2d34f698a5004a36e66b78562418ef3c1db1226fb1d970b94f6898c7c"} Oct 30 18:28:27.867196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.867116 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8be79fb2d34f698a5004a36e66b78562418ef3c1db1226fb1d970b94f6898c7c" Oct 30 18:28:27.966996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:27.966936 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:28.614201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:28.614164 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:28.795878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:28.795813 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:28.849668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:28.849641 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2"] Oct 30 18:28:28.859177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:28.859149 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a4f3f0-gfbk2"] Oct 30 18:28:28.956286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:28.956243 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:29.301292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:29.301227 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b93d5d5-ca47-42a3-b018-ba1908b65eba" path="/var/lib/kubelet/pods/3b93d5d5-ca47-42a3-b018-ba1908b65eba/volumes" Oct 30 18:28:29.329434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:29.329407 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:29.998733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:29.998701 4582 scope.go:122] "RemoveContainer" containerID="a43bfb1f9b2453f15013d2b4056300f8edffcfa8b9c537b03b8b4365eeb96735" Oct 30 18:28:30.152586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.152554 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" podUID="32d336ca-3ee7-42d3-855a-a06662e15ebc" containerName="payload-processing" containerID="cri-o://c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a" gracePeriod=25 Oct 30 18:28:30.193815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.193781 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" podUID="ac44c39c-548c-4ab7-a6f2-8f726b047d38" containerName="payload-pre-processing" containerID="cri-o://ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3" gracePeriod=25 Oct 30 18:28:30.465541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.465512 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:28:30.487677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.487654 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:28:30.576100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576028 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac44c39c-548c-4ab7-a6f2-8f726b047d38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac44c39c-548c-4ab7-a6f2-8f726b047d38-plugins-config-volume\") pod \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " Oct 30 18:28:30.576100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576070 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac44c39c-548c-4ab7-a6f2-8f726b047d38-kube-api-access-xqgqw\" (UniqueName: \"kubernetes.io/projected/ac44c39c-548c-4ab7-a6f2-8f726b047d38-kube-api-access-xqgqw\") pod \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\" (UID: \"ac44c39c-548c-4ab7-a6f2-8f726b047d38\") " Oct 30 18:28:30.576327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576162 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d336ca-3ee7-42d3-855a-a06662e15ebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d336ca-3ee7-42d3-855a-a06662e15ebc-plugins-config-volume\") pod \"32d336ca-3ee7-42d3-855a-a06662e15ebc\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " Oct 30 18:28:30.576327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576205 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d336ca-3ee7-42d3-855a-a06662e15ebc-kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/32d336ca-3ee7-42d3-855a-a06662e15ebc-kube-api-access-gvp6m\") pod \"32d336ca-3ee7-42d3-855a-a06662e15ebc\" (UID: \"32d336ca-3ee7-42d3-855a-a06662e15ebc\") " Oct 30 18:28:30.576484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576454 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac44c39c-548c-4ab7-a6f2-8f726b047d38-plugins-config-volume" pod "ac44c39c-548c-4ab7-a6f2-8f726b047d38" (UID: "ac44c39c-548c-4ab7-a6f2-8f726b047d38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:30.576542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576511 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d336ca-3ee7-42d3-855a-a06662e15ebc-plugins-config-volume" pod "32d336ca-3ee7-42d3-855a-a06662e15ebc" (UID: "32d336ca-3ee7-42d3-855a-a06662e15ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:30.576650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576634 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac44c39c-548c-4ab7-a6f2-8f726b047d38-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:30.576710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.576652 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d336ca-3ee7-42d3-855a-a06662e15ebc-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:30.578160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.578137 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac44c39c-548c-4ab7-a6f2-8f726b047d38-kube-api-access-xqgqw" pod "ac44c39c-548c-4ab7-a6f2-8f726b047d38" (UID: "ac44c39c-548c-4ab7-a6f2-8f726b047d38"). InnerVolumeSpecName "kube-api-access-xqgqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:30.578270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.578208 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d336ca-3ee7-42d3-855a-a06662e15ebc-kube-api-access-gvp6m" pod "32d336ca-3ee7-42d3-855a-a06662e15ebc" (UID: "32d336ca-3ee7-42d3-855a-a06662e15ebc"). InnerVolumeSpecName "kube-api-access-gvp6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:30.677453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.677429 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/32d336ca-3ee7-42d3-855a-a06662e15ebc-kube-api-access-gvp6m\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:30.677453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.677455 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqgqw\" (UniqueName: \"kubernetes.io/projected/ac44c39c-548c-4ab7-a6f2-8f726b047d38-kube-api-access-xqgqw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:30.891184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.891148 4582 generic.go:356] "Generic (PLEG): container finished" podID="ac44c39c-548c-4ab7-a6f2-8f726b047d38" containerID="ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3" exitCode=0 Oct 30 18:28:30.891368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.891199 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" Oct 30 18:28:30.891368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.891230 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" event={"ID":"ac44c39c-548c-4ab7-a6f2-8f726b047d38","Type":"ContainerDied","Data":"ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3"} Oct 30 18:28:30.891368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.891287 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl" event={"ID":"ac44c39c-548c-4ab7-a6f2-8f726b047d38","Type":"ContainerDied","Data":"cddbcb6f87e60f74e207666a0abd6c38f6975584ad7632fb2e665253bd0fa438"} Oct 30 18:28:30.891368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.891313 4582 scope.go:122] "RemoveContainer" containerID="ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3" Oct 30 18:28:30.892901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.892878 4582 generic.go:356] "Generic (PLEG): container finished" podID="32d336ca-3ee7-42d3-855a-a06662e15ebc" containerID="c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a" exitCode=0 Oct 30 18:28:30.893043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.892922 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" Oct 30 18:28:30.893043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.892969 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" event={"ID":"32d336ca-3ee7-42d3-855a-a06662e15ebc","Type":"ContainerDied","Data":"c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a"} Oct 30 18:28:30.893043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.893019 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7" event={"ID":"32d336ca-3ee7-42d3-855a-a06662e15ebc","Type":"ContainerDied","Data":"c748d506598c74cffb1b2f92db825dc8b1009dd84c3018c41acec018c11d38f1"} Oct 30 18:28:30.902113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.902013 4582 scope.go:122] "RemoveContainer" containerID="ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3" Oct 30 18:28:30.902394 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:30.902367 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3\": container with ID starting with ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3 not found: ID does not exist" containerID="ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3" Oct 30 18:28:30.902465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.902401 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3"} err="failed to get container status \"ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3\": rpc error: code = NotFound desc = could not find container \"ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3\": container with ID starting with ecfb65676ffc586a00ce3e07a7f39136b60013c2dd552faa64697c36a07767a3 not found: ID does not exist" Oct 30 18:28:30.902465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.902424 4582 scope.go:122] "RemoveContainer" containerID="c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a" Oct 30 18:28:30.911879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.911861 4582 scope.go:122] "RemoveContainer" containerID="c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a" Oct 30 18:28:30.912171 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:30.912154 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a\": container with ID starting with c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a not found: ID does not exist" containerID="c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a" Oct 30 18:28:30.912225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.912177 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a"} err="failed to get container status \"c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a\": rpc error: code = NotFound desc = could not find container \"c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a\": container with ID starting with c0dc8c72c243543b3e88a546aa0a74552b66da1818cdbfa26a288280719e761a not found: ID does not exist" Oct 30 18:28:30.935639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.935606 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl"] Oct 30 18:28:30.938595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.938573 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-44379a-79d8645f67-llqwl"] Oct 30 18:28:30.968872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.968841 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7"] Oct 30 18:28:30.972922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.972893 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-44379a-6c575b68f4-pkdt7"] Oct 30 18:28:30.973052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:30.972975 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:31.300531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:31.300501 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d336ca-3ee7-42d3-855a-a06662e15ebc" path="/var/lib/kubelet/pods/32d336ca-3ee7-42d3-855a-a06662e15ebc/volumes" Oct 30 18:28:31.300913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:31.300904 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac44c39c-548c-4ab7-a6f2-8f726b047d38" path="/var/lib/kubelet/pods/ac44c39c-548c-4ab7-a6f2-8f726b047d38/volumes" Oct 30 18:28:32.784884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:32.784852 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:33.979080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:33.979050 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr"] Oct 30 18:28:34.010241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.010212 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662"] Oct 30 18:28:34.104341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.104309 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.204655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.204623 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.216368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.215653 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67"] Oct 30 18:28:34.216786 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.216748 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" podUID="1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" containerName="maas-api" containerID="cri-o://53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b" gracePeriod=30 Oct 30 18:28:34.290254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.290221 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" podUID="1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.159:8443/health\": dial tcp 10.132.0.159:8443: connect: connection refused" Oct 30 18:28:34.305540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.305509 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.404782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.404748 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.483533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.483478 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:28:34.523168 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.523142 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-openshift-service-ca\") pod \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " Oct 30 18:28:34.523321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.523249 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-kube-api-access-znsvn\" (UniqueName: \"kubernetes.io/projected/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-kube-api-access-znsvn\") pod \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " Oct 30 18:28:34.523369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.523326 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-metrics-certs\") pod \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " Oct 30 18:28:34.523369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.523354 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-maas-api-tls\") pod \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\" (UID: \"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0\") " Oct 30 18:28:34.523502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.523480 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-openshift-service-ca" pod "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" (UID: "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:34.523858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.523836 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.525450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.525423 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-kube-api-access-znsvn" pod "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" (UID: "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0"). InnerVolumeSpecName "kube-api-access-znsvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:34.525649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.525621 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-maas-api-tls" pod "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" (UID: "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:34.525889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.525860 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-metrics-certs" pod "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" (UID: "1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:34.625358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.625309 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znsvn\" (UniqueName: \"kubernetes.io/projected/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-kube-api-access-znsvn\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.625358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.625350 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.625358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.625364 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.743463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.743194 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.925197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.925161 4582 generic.go:356] "Generic (PLEG): container finished" podID="1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" containerID="53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b" exitCode=0 Oct 30 18:28:34.925348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.925203 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" Oct 30 18:28:34.925348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.925241 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" event={"ID":"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0","Type":"ContainerDied","Data":"53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b"} Oct 30 18:28:34.925348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.925294 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67" event={"ID":"1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0","Type":"ContainerDied","Data":"527d9e9c2ec5995fbacae4d71ce7adcead958394fbfda8844d3555dd8260465c"} Oct 30 18:28:34.925348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.925321 4582 scope.go:122] "RemoveContainer" containerID="53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b" Oct 30 18:28:34.935856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.935835 4582 scope.go:122] "RemoveContainer" containerID="53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b" Oct 30 18:28:34.936141 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:34.936123 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b\": container with ID starting with 53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b not found: ID does not exist" containerID="53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b" Oct 30 18:28:34.936185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.936149 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b"} err="failed to get container status \"53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b\": rpc error: code = NotFound desc = could not find container \"53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b\": container with ID starting with 53b8df8a4fb6aede35d6ac1bf0c1ea22ba4bb333f73349874d3cf52af2eb4c3b not found: ID does not exist" Oct 30 18:28:34.969382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.969353 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67"] Oct 30 18:28:34.978479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:34.978457 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-42bf6e-76779b4c87-qgf67"] Oct 30 18:28:35.084563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.084486 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:35.185661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.185628 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:35.308755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.308720 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" path="/var/lib/kubelet/pods/1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0/volumes" Oct 30 18:28:35.544052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.544015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:35.818463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.818370 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:35.843813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.843777 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n"] Oct 30 18:28:35.847348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.847326 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42bf6e-k8k5n"] Oct 30 18:28:35.909318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:35.909284 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:36.067090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:36.067059 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:36.178208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:36.178175 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:36.225679 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:36.225647 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:28:36.241573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:36.241532 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz"] Oct 30 18:28:36.244507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:36.244479 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f826dd-vqfbz"] Oct 30 18:28:36.286030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:36.285996 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:37.300267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:37.300232 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" path="/var/lib/kubelet/pods/d11f8d2b-c019-4fbf-be90-b679fbdbfdcb/volumes" Oct 30 18:28:37.300674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:37.300643 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4111b3e-d0fc-42f2-932f-db9ba41043d3" path="/var/lib/kubelet/pods/f4111b3e-d0fc-42f2-932f-db9ba41043d3/volumes" Oct 30 18:28:37.763968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:37.763937 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:38.210977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.210941 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th"] Oct 30 18:28:38.211490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211478 4582 cpu_manager.go:414] "containerMap: removing container" podUID="1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" containerName="maas-api" Oct 30 18:28:38.211549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211492 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" containerName="maas-api" Oct 30 18:28:38.211549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211505 4582 cpu_manager.go:414] "containerMap: removing container" podUID="ac44c39c-548c-4ab7-a6f2-8f726b047d38" containerName="payload-pre-processing" Oct 30 18:28:38.211549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211510 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac44c39c-548c-4ab7-a6f2-8f726b047d38" containerName="payload-pre-processing" Oct 30 18:28:38.211549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211522 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" containerName="revoke-keys" Oct 30 18:28:38.211549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211527 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" containerName="revoke-keys" Oct 30 18:28:38.211549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211546 4582 cpu_manager.go:414] "containerMap: removing container" podUID="32d336ca-3ee7-42d3-855a-a06662e15ebc" containerName="payload-processing" Oct 30 18:28:38.211549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211552 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d336ca-3ee7-42d3-855a-a06662e15ebc" containerName="payload-processing" Oct 30 18:28:38.211781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211657 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d336ca-3ee7-42d3-855a-a06662e15ebc" containerName="payload-processing" Oct 30 18:28:38.211781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211676 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c93c5c8-d95e-4dc7-ac89-8e5c87ce42f0" containerName="maas-api" Oct 30 18:28:38.211781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211689 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac44c39c-548c-4ab7-a6f2-8f726b047d38" containerName="payload-pre-processing" Oct 30 18:28:38.211781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.211701 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11f8d2b-c019-4fbf-be90-b679fbdbfdcb" containerName="revoke-keys" Oct 30 18:28:38.216678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.216652 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.235630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.235609 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv"] Oct 30 18:28:38.239513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.239498 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.241609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.241595 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-921450-openshift-default-dockercfg-rt5dw\"" Oct 30 18:28:38.251627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.251606 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th"] Oct 30 18:28:38.271263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.271235 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv"] Oct 30 18:28:38.279348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279315 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.279488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279355 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.279564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279488 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.279628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279614 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.279725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279683 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xf5\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-kube-api-access-p2xf5\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.279889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279721 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.279889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279766 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.279889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279865 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.280023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.279930 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.381418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381383 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381425 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381447 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381534 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381576 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381606 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381692 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381735 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xf5\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-kube-api-access-p2xf5\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381841 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.381898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381861 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381947 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.381987 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382036 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6lp\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-kube-api-access-8c6lp\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382065 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382081 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382255 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382294 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382323 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.382392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382349 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382408 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382563 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.382717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382573 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.382717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.382671 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.383946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.383918 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.384612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.384597 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.397859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.397839 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.398076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.398060 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xf5\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-kube-api-access-p2xf5\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.484297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484200 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484262 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484325 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484357 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484399 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6lp\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-kube-api-access-8c6lp\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484430 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484499 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484544 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484589 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484706 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.484927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484786 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.485098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.484965 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.485098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.485026 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.485098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.485061 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.486732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.486710 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.486899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.486881 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.500462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.500433 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.500653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.500634 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6lp\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-kube-api-access-8c6lp\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.531511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.531484 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:38.555262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.555237 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:38.850582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.850544 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th"] Oct 30 18:28:38.856422 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:38.856382 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284b15a6_7a92_4062_8aa2_44b6c4ed9f79.slice/crio-0acd1534e92f8bacb2fa904856cdb7c46ba8982163573d3d468e6e5459e6f8df WatchSource:0}: Error finding container 0acd1534e92f8bacb2fa904856cdb7c46ba8982163573d3d468e6e5459e6f8df: Status 404 returned error can't find the container with id 0acd1534e92f8bacb2fa904856cdb7c46ba8982163573d3d468e6e5459e6f8df Oct 30 18:28:38.858650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.858615 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:28:38.858743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.858693 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:28:38.858743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.858737 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:28:38.883521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.883485 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv"] Oct 30 18:28:38.888251 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:38.888219 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c107704_6d25_42cd_ab4d_b1c69eeb7759.slice/crio-ab0e772b1bb5a68faa74d87cfde9b4ffd164aa5a5b8d19ac977b66c77d279d9e WatchSource:0}: Error finding container ab0e772b1bb5a68faa74d87cfde9b4ffd164aa5a5b8d19ac977b66c77d279d9e: Status 404 returned error can't find the container with id ab0e772b1bb5a68faa74d87cfde9b4ffd164aa5a5b8d19ac977b66c77d279d9e Oct 30 18:28:38.890798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.890768 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:28:38.890885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.890853 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:28:38.890925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.890887 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:28:38.959330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.958617 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" event={"ID":"2c107704-6d25-42cd-ab4d-b1c69eeb7759","Type":"ContainerStarted","Data":"b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb"} Oct 30 18:28:38.959330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.958675 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" event={"ID":"2c107704-6d25-42cd-ab4d-b1c69eeb7759","Type":"ContainerStarted","Data":"ab0e772b1bb5a68faa74d87cfde9b4ffd164aa5a5b8d19ac977b66c77d279d9e"} Oct 30 18:28:38.960487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.960455 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" event={"ID":"284b15a6-7a92-4062-8aa2-44b6c4ed9f79","Type":"ContainerStarted","Data":"2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9"} Oct 30 18:28:38.960632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.960508 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" event={"ID":"284b15a6-7a92-4062-8aa2-44b6c4ed9f79","Type":"ContainerStarted","Data":"0acd1534e92f8bacb2fa904856cdb7c46ba8982163573d3d468e6e5459e6f8df"} Oct 30 18:28:38.979954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:38.979804 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" podUID="ef958611-b15d-4654-9144-1c15388dbef3" containerName="payload-processing" containerID="cri-o://29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf" gracePeriod=25 Oct 30 18:28:39.001848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.000344 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podStartSLOduration=1.000323986 podStartE2EDuration="1.000323986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:38.997919245 +0000 UTC m=+1815.822313541" watchObservedRunningTime="2026-10-30 18:28:39.000323986 +0000 UTC m=+1815.824718304" Oct 30 18:28:39.010545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.010481 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" podUID="dc5c594e-4039-47db-9533-c5db3b98ddba" containerName="payload-pre-processing" containerID="cri-o://3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a" gracePeriod=25 Oct 30 18:28:39.036270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.036160 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podStartSLOduration=1.036145377 podStartE2EDuration="1.036145377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:39.03263054 +0000 UTC m=+1815.857024832" watchObservedRunningTime="2026-10-30 18:28:39.036145377 +0000 UTC m=+1815.860539669" Oct 30 18:28:39.285454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.285425 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:28:39.337994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.337961 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:28:39.399520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.399492 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef958611-b15d-4654-9144-1c15388dbef3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef958611-b15d-4654-9144-1c15388dbef3-plugins-config-volume\") pod \"ef958611-b15d-4654-9144-1c15388dbef3\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " Oct 30 18:28:39.399950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.399660 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5c594e-4039-47db-9533-c5db3b98ddba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5c594e-4039-47db-9533-c5db3b98ddba-plugins-config-volume\") pod \"dc5c594e-4039-47db-9533-c5db3b98ddba\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " Oct 30 18:28:39.399950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.399716 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5c594e-4039-47db-9533-c5db3b98ddba-kube-api-access-mj2pb\" (UniqueName: \"kubernetes.io/projected/dc5c594e-4039-47db-9533-c5db3b98ddba-kube-api-access-mj2pb\") pod \"dc5c594e-4039-47db-9533-c5db3b98ddba\" (UID: \"dc5c594e-4039-47db-9533-c5db3b98ddba\") " Oct 30 18:28:39.399950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.399781 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef958611-b15d-4654-9144-1c15388dbef3-kube-api-access-87jcm\" (UniqueName: \"kubernetes.io/projected/ef958611-b15d-4654-9144-1c15388dbef3-kube-api-access-87jcm\") pod \"ef958611-b15d-4654-9144-1c15388dbef3\" (UID: \"ef958611-b15d-4654-9144-1c15388dbef3\") " Oct 30 18:28:39.400089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.399960 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef958611-b15d-4654-9144-1c15388dbef3-plugins-config-volume" pod "ef958611-b15d-4654-9144-1c15388dbef3" (UID: "ef958611-b15d-4654-9144-1c15388dbef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:39.400581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.400462 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef958611-b15d-4654-9144-1c15388dbef3-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:39.400934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.400895 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5c594e-4039-47db-9533-c5db3b98ddba-plugins-config-volume" pod "dc5c594e-4039-47db-9533-c5db3b98ddba" (UID: "dc5c594e-4039-47db-9533-c5db3b98ddba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:39.402507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.402485 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef958611-b15d-4654-9144-1c15388dbef3-kube-api-access-87jcm" pod "ef958611-b15d-4654-9144-1c15388dbef3" (UID: "ef958611-b15d-4654-9144-1c15388dbef3"). InnerVolumeSpecName "kube-api-access-87jcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:39.402680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.402658 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5c594e-4039-47db-9533-c5db3b98ddba-kube-api-access-mj2pb" pod "dc5c594e-4039-47db-9533-c5db3b98ddba" (UID: "dc5c594e-4039-47db-9533-c5db3b98ddba"). InnerVolumeSpecName "kube-api-access-mj2pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:39.501956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.501928 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5c594e-4039-47db-9533-c5db3b98ddba-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:39.501956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.501954 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2pb\" (UniqueName: \"kubernetes.io/projected/dc5c594e-4039-47db-9533-c5db3b98ddba-kube-api-access-mj2pb\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:39.502157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.501964 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87jcm\" (UniqueName: \"kubernetes.io/projected/ef958611-b15d-4654-9144-1c15388dbef3-kube-api-access-87jcm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:39.531849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.531792 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:39.533411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.533386 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:39.533515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.533443 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 18:28:39.556047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.556022 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:39.557231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.557209 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:28:39.557387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.557255 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 18:28:39.643423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.643394 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:39.969223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.969178 4582 generic.go:356] "Generic (PLEG): container finished" podID="dc5c594e-4039-47db-9533-c5db3b98ddba" containerID="3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a" exitCode=0 Oct 30 18:28:39.969223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.969222 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" Oct 30 18:28:39.969508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.969255 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" event={"ID":"dc5c594e-4039-47db-9533-c5db3b98ddba","Type":"ContainerDied","Data":"3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a"} Oct 30 18:28:39.969508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.969318 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662" event={"ID":"dc5c594e-4039-47db-9533-c5db3b98ddba","Type":"ContainerDied","Data":"b8abe9bb900731f6f4d54b6dbf2e103abe7c9db4d2f229244d862dd4d7fd6df8"} Oct 30 18:28:39.969508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.969350 4582 scope.go:122] "RemoveContainer" containerID="3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a" Oct 30 18:28:39.971673 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.971641 4582 generic.go:356] "Generic (PLEG): container finished" podID="ef958611-b15d-4654-9144-1c15388dbef3" containerID="29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf" exitCode=0 Oct 30 18:28:39.971811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.971746 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" Oct 30 18:28:39.971913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.971889 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" event={"ID":"ef958611-b15d-4654-9144-1c15388dbef3","Type":"ContainerDied","Data":"29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf"} Oct 30 18:28:39.972003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.971970 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr" event={"ID":"ef958611-b15d-4654-9144-1c15388dbef3","Type":"ContainerDied","Data":"af3c700fd0009b99d945e67907766323f2256d866eac6802e41a0951b84fa3c0"} Oct 30 18:28:39.986652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.986630 4582 scope.go:122] "RemoveContainer" containerID="3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a" Oct 30 18:28:39.986979 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:39.986956 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a\": container with ID starting with 3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a not found: ID does not exist" containerID="3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a" Oct 30 18:28:39.987114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.986989 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a"} err="failed to get container status \"3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a\": rpc error: code = NotFound desc = could not find container \"3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a\": container with ID starting with 3aaed8d24b73a0cea9065680b8ac5a43c64a6cc74c8bfafbb2a366258819658a not found: ID does not exist" Oct 30 18:28:39.987114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:39.987012 4582 scope.go:122] "RemoveContainer" containerID="29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf" Oct 30 18:28:40.008024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.007982 4582 scope.go:122] "RemoveContainer" containerID="29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf" Oct 30 18:28:40.008390 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:28:40.008367 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf\": container with ID starting with 29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf not found: ID does not exist" containerID="29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf" Oct 30 18:28:40.008532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.008400 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf"} err="failed to get container status \"29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf\": rpc error: code = NotFound desc = could not find container \"29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf\": container with ID starting with 29d5600cb12d8a177745cfe4666beccbbd021ca3f225d4545f7ebf55b4cc74cf not found: ID does not exist" Oct 30 18:28:40.032689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.032646 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662"] Oct 30 18:28:40.041135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.041105 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-42bf6e-658ff5475-st662"] Oct 30 18:28:40.093616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.093581 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr"] Oct 30 18:28:40.093874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.093855 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-42bf6e-69fdcc7bcb-4q4fr"] Oct 30 18:28:40.532703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.532667 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:40.533218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.532723 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 18:28:40.575846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.574970 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:28:40.575846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:40.575096 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 18:28:41.002775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.002722 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:41.019461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.019181 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb"] Oct 30 18:28:41.019871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.019854 4582 cpu_manager.go:414] "containerMap: removing container" podUID="dc5c594e-4039-47db-9533-c5db3b98ddba" containerName="payload-pre-processing" Oct 30 18:28:41.020050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.019873 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5c594e-4039-47db-9533-c5db3b98ddba" containerName="payload-pre-processing" Oct 30 18:28:41.020050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.019891 4582 cpu_manager.go:414] "containerMap: removing container" podUID="ef958611-b15d-4654-9144-1c15388dbef3" containerName="payload-processing" Oct 30 18:28:41.020050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.019896 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef958611-b15d-4654-9144-1c15388dbef3" containerName="payload-processing" Oct 30 18:28:41.020050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.020001 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5c594e-4039-47db-9533-c5db3b98ddba" containerName="payload-pre-processing" Oct 30 18:28:41.020050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.020013 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef958611-b15d-4654-9144-1c15388dbef3" containerName="payload-processing" Oct 30 18:28:41.024941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.024923 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.039653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.039627 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb"] Oct 30 18:28:41.067849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.053057 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:28:41.067849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.053135 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:41.126745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.122493 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvpt\" (UniqueName: \"kubernetes.io/projected/428d12ed-2e29-4d01-a648-bdb392bee574-kube-api-access-pwvpt\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.126745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.122593 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.126745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.123228 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.220567 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.220526 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:41.225604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.225568 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.225769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.225755 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.225899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.225881 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvpt\" (UniqueName: \"kubernetes.io/projected/428d12ed-2e29-4d01-a648-bdb392bee574-kube-api-access-pwvpt\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.226941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.226917 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.231931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.231779 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.284902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.247482 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvpt\" (UniqueName: \"kubernetes.io/projected/428d12ed-2e29-4d01-a648-bdb392bee574-kube-api-access-pwvpt\") pod \"maas-api-revoke-sub-021a7a92-f8gnb\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.310838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.310799 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5c594e-4039-47db-9533-c5db3b98ddba" path="/var/lib/kubelet/pods/dc5c594e-4039-47db-9533-c5db3b98ddba/volumes" Oct 30 18:28:41.311226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.311208 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef958611-b15d-4654-9144-1c15388dbef3" path="/var/lib/kubelet/pods/ef958611-b15d-4654-9144-1c15388dbef3/volumes" Oct 30 18:28:41.338439 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.338416 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:41.532221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.532182 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:41.532395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.532265 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 18:28:41.566901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.566282 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:28:41.566901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.566344 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 18:28:41.897813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:41.896703 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb"] Oct 30 18:28:41.901122 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:41.901091 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428d12ed_2e29_4d01_a648_bdb392bee574.slice/crio-5243600e18cb06d7fec282587d7bae5e77195b6b24d495b08dc55540b15a4317 WatchSource:0}: Error finding container 5243600e18cb06d7fec282587d7bae5e77195b6b24d495b08dc55540b15a4317: Status 404 returned error can't find the container with id 5243600e18cb06d7fec282587d7bae5e77195b6b24d495b08dc55540b15a4317 Oct 30 18:28:42.020596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.020550 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" event={"ID":"428d12ed-2e29-4d01-a648-bdb392bee574","Type":"ContainerStarted","Data":"5243600e18cb06d7fec282587d7bae5e77195b6b24d495b08dc55540b15a4317"} Oct 30 18:28:42.387840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.372754 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.513959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.513777 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.536256 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.536222 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:42.536524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.536494 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 18:28:42.560913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.560880 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:28:42.561187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.561157 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 18:28:42.614086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.614048 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.911991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.911955 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.912364 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.912274 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb"] Oct 30 18:28:42.913050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:42.913023 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" containerID="cri-o://74d154ff598ef652f42446ce055b19d8a513dfd8cb0818b5ca0b36a4685f544b" gracePeriod=30 Oct 30 18:28:43.050549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:43.050503 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" event={"ID":"428d12ed-2e29-4d01-a648-bdb392bee574","Type":"ContainerStarted","Data":"b56a535a6eb3d5c2f65ac790295a4f17ebd2c2975e9cf5cb668df439d984486e"} Oct 30 18:28:43.108500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:43.108349 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" podStartSLOduration=3.108327802 podStartE2EDuration="3.108327802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:43.106483847 +0000 UTC m=+1819.930878140" watchObservedRunningTime="2026-10-30 18:28:43.108327802 +0000 UTC m=+1819.932722097" Oct 30 18:28:43.532508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:43.532471 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:43.532688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:43.532522 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 18:28:43.566844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:43.565930 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:28:43.566844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:43.565995 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 18:28:44.095455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:44.095423 4582 generic.go:356] "Generic (PLEG): container finished" podID="428d12ed-2e29-4d01-a648-bdb392bee574" containerID="b56a535a6eb3d5c2f65ac790295a4f17ebd2c2975e9cf5cb668df439d984486e" exitCode=0 Oct 30 18:28:44.095916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:44.095524 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" event={"ID":"428d12ed-2e29-4d01-a648-bdb392bee574","Type":"ContainerDied","Data":"b56a535a6eb3d5c2f65ac790295a4f17ebd2c2975e9cf5cb668df439d984486e"} Oct 30 18:28:44.369904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:44.367963 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:44.532242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:44.532200 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:44.532417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:44.532265 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 18:28:44.562844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:44.562193 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:28:44.562844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:44.562251 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 18:28:45.030358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.030314 4582 patch_prober.go:29] interesting pod/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:28:45.030533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.030383 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:28:45.335681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.335651 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:45.466311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.466266 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-xks-ca\") pod \"428d12ed-2e29-4d01-a648-bdb392bee574\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " Oct 30 18:28:45.467137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.467113 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428d12ed-2e29-4d01-a648-bdb392bee574-kube-api-access-pwvpt\" (UniqueName: \"kubernetes.io/projected/428d12ed-2e29-4d01-a648-bdb392bee574-kube-api-access-pwvpt\") pod \"428d12ed-2e29-4d01-a648-bdb392bee574\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " Oct 30 18:28:45.475046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.474996 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-openshift-ca" pod "428d12ed-2e29-4d01-a648-bdb392bee574" (UID: "428d12ed-2e29-4d01-a648-bdb392bee574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:45.475405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.475383 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-openshift-ca\") pod \"428d12ed-2e29-4d01-a648-bdb392bee574\" (UID: \"428d12ed-2e29-4d01-a648-bdb392bee574\") " Oct 30 18:28:45.496281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.496162 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428d12ed-2e29-4d01-a648-bdb392bee574-kube-api-access-pwvpt" pod "428d12ed-2e29-4d01-a648-bdb392bee574" (UID: "428d12ed-2e29-4d01-a648-bdb392bee574"). InnerVolumeSpecName "kube-api-access-pwvpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:45.496456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.496313 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-xks-ca" pod "428d12ed-2e29-4d01-a648-bdb392bee574" (UID: "428d12ed-2e29-4d01-a648-bdb392bee574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:45.532781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.532750 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:28:45.532920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.532805 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 18:28:45.556166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.556058 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 18:28:45.556166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.556115 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 18:28:45.577903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.577867 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvpt\" (UniqueName: \"kubernetes.io/projected/428d12ed-2e29-4d01-a648-bdb392bee574-kube-api-access-pwvpt\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:45.577903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.577895 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:45.577903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:45.577906 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428d12ed-2e29-4d01-a648-bdb392bee574-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:46.113900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.113864 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" event={"ID":"428d12ed-2e29-4d01-a648-bdb392bee574","Type":"ContainerDied","Data":"5243600e18cb06d7fec282587d7bae5e77195b6b24d495b08dc55540b15a4317"} Oct 30 18:28:46.113900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.113881 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb" Oct 30 18:28:46.114181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.113917 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5243600e18cb06d7fec282587d7bae5e77195b6b24d495b08dc55540b15a4317" Oct 30 18:28:46.265138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.265108 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:46.726806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.726765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:46.924618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.924587 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:46.925110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.925085 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:46.925923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.925905 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:28:46.939213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.939191 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:46.939735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.939708 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:46.940521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:46.940499 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:28:48.141259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.140706 4582 generic.go:356] "Generic (PLEG): container finished" podID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerID="74d154ff598ef652f42446ce055b19d8a513dfd8cb0818b5ca0b36a4685f544b" exitCode=0 Oct 30 18:28:48.141259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.141092 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" event={"ID":"9defb28d-0725-409e-80d5-ba3b0df5d4fd","Type":"ContainerDied","Data":"74d154ff598ef652f42446ce055b19d8a513dfd8cb0818b5ca0b36a4685f544b"} Oct 30 18:28:48.143163 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.143139 4582 scope.go:122] "RemoveContainer" containerID="4ef30f742dcc2cd71f2c09724d0154063f9542b7730e02c11588f9d357efb16e" Oct 30 18:28:48.231082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.231055 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:28:48.330022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.329990 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-envoy\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330033 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-socket\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330055 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-data\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330084 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-credential-socket\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330105 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istiod-ca-cert\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330168 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-podinfo\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330245 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-certs\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330301 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-token\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-token\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330325 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-kube-api-access-t5ckc\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-kube-api-access-t5ckc\") pod \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\" (UID: \"9defb28d-0725-409e-80d5-ba3b0df5d4fd\") " Oct 30 18:28:48.330517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330336 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-socket" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.330517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330501 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-credential-socket" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.330958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330515 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-certs" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.330958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330597 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istiod-ca-cert" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:48.330958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.330899 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-data" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.331313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.331292 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.331391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.331320 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.331391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.331336 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.331391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.331353 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.331391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.331368 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.332480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.332447 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-podinfo" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:48.332602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.332498 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-envoy" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.332602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.332551 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-token" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:48.332890 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.332861 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-kube-api-access-t5ckc" pod "9defb28d-0725-409e-80d5-ba3b0df5d4fd" (UID: "9defb28d-0725-409e-80d5-ba3b0df5d4fd"). InnerVolumeSpecName "kube-api-access-t5ckc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:48.433068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.433030 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.433068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.433065 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5ckc\" (UniqueName: \"kubernetes.io/projected/9defb28d-0725-409e-80d5-ba3b0df5d4fd-kube-api-access-t5ckc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.433068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.433076 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.433291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.433086 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9defb28d-0725-409e-80d5-ba3b0df5d4fd-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.833600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:48.833564 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:49.149271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:49.149249 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" Oct 30 18:28:49.149717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:49.149292 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb" event={"ID":"9defb28d-0725-409e-80d5-ba3b0df5d4fd","Type":"ContainerDied","Data":"4d8aa523ccc70909c95e3a1bad356d5af4a756f8932bb0b3b0cd80460f3eff44"} Oct 30 18:28:49.149717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:49.149359 4582 scope.go:122] "RemoveContainer" containerID="74d154ff598ef652f42446ce055b19d8a513dfd8cb0818b5ca0b36a4685f544b" Oct 30 18:28:49.203886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:49.203862 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb"] Oct 30 18:28:49.209804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:49.209784 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-42bf6e-openshift-default-6bc8854cb5-vnqpb"] Oct 30 18:28:49.300757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:49.300728 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" path="/var/lib/kubelet/pods/9defb28d-0725-409e-80d5-ba3b0df5d4fd/volumes" Oct 30 18:28:50.611630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:50.611596 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:52.411129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:52.411099 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:52.559989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:52.559954 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:52.660910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:52.660874 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:53.967956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.967912 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4"] Oct 30 18:28:53.968477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968463 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" Oct 30 18:28:53.968534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968478 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" Oct 30 18:28:53.968534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968494 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" Oct 30 18:28:53.968534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968500 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" Oct 30 18:28:53.968534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968520 4582 cpu_manager.go:414] "containerMap: removing container" podUID="428d12ed-2e29-4d01-a648-bdb392bee574" containerName="revoke-keys" Oct 30 18:28:53.968534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968525 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428d12ed-2e29-4d01-a648-bdb392bee574" containerName="revoke-keys" Oct 30 18:28:53.968737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968614 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" Oct 30 18:28:53.968737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968623 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9defb28d-0725-409e-80d5-ba3b0df5d4fd" containerName="istio-proxy" Oct 30 18:28:53.968737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.968632 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="428d12ed-2e29-4d01-a648-bdb392bee574" containerName="revoke-keys" Oct 30 18:28:53.973038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.973021 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:53.996420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.996389 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:53.996535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.996518 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:28:53.998800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.998763 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:53.999017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.998993 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4"] Oct 30 18:28:53.999146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.999096 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8z9\" (UniqueName: \"kubernetes.io/projected/e26cc1c5-716e-4551-b853-9beed051b54c-kube-api-access-jj8z9\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:53.999219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:53.999182 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.100855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.100770 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8z9\" (UniqueName: \"kubernetes.io/projected/e26cc1c5-716e-4551-b853-9beed051b54c-kube-api-access-jj8z9\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.101072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.100956 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.101072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.101054 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.101631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.101610 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.103321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.103302 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.120229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.120202 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8z9\" (UniqueName: \"kubernetes.io/projected/e26cc1c5-716e-4551-b853-9beed051b54c-kube-api-access-jj8z9\") pod \"maas-api-revoke-sub-c79a68bc-wlkg4\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.165133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.165100 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:54.287880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.287767 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:54.588118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:54.588037 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4"] Oct 30 18:28:54.591261 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:28:54.591228 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26cc1c5_716e_4551_b853_9beed051b54c.slice/crio-2556b1584269b5be11a4c51cb13b35d4025d7094abf2f5ea912acdb0a766110f WatchSource:0}: Error finding container 2556b1584269b5be11a4c51cb13b35d4025d7094abf2f5ea912acdb0a766110f: Status 404 returned error can't find the container with id 2556b1584269b5be11a4c51cb13b35d4025d7094abf2f5ea912acdb0a766110f Oct 30 18:28:55.191212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:55.191171 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" event={"ID":"e26cc1c5-716e-4551-b853-9beed051b54c","Type":"ContainerStarted","Data":"dda6810e8ec532048e824565e75eb2f5bfeb06a205b6622d0c0d56d2223ec000"} Oct 30 18:28:55.191615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:55.191232 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" event={"ID":"e26cc1c5-716e-4551-b853-9beed051b54c","Type":"ContainerStarted","Data":"2556b1584269b5be11a4c51cb13b35d4025d7094abf2f5ea912acdb0a766110f"} Oct 30 18:28:55.226347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:55.226299 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" podStartSLOduration=2.226285801 podStartE2EDuration="2.226285801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:55.225247549 +0000 UTC m=+1832.049641841" watchObservedRunningTime="2026-10-30 18:28:55.226285801 +0000 UTC m=+1832.050680093" Oct 30 18:28:55.983523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:55.983489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:56.199458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:56.199426 4582 generic.go:356] "Generic (PLEG): container finished" podID="e26cc1c5-716e-4551-b853-9beed051b54c" containerID="dda6810e8ec532048e824565e75eb2f5bfeb06a205b6622d0c0d56d2223ec000" exitCode=0 Oct 30 18:28:56.199865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:56.199513 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" event={"ID":"e26cc1c5-716e-4551-b853-9beed051b54c","Type":"ContainerDied","Data":"dda6810e8ec532048e824565e75eb2f5bfeb06a205b6622d0c0d56d2223ec000"} Oct 30 18:28:57.357262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.357238 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:57.438518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.438490 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26cc1c5-716e-4551-b853-9beed051b54c-kube-api-access-jj8z9\" (UniqueName: \"kubernetes.io/projected/e26cc1c5-716e-4551-b853-9beed051b54c-kube-api-access-jj8z9\") pod \"e26cc1c5-716e-4551-b853-9beed051b54c\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " Oct 30 18:28:57.438702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.438541 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-xks-ca\") pod \"e26cc1c5-716e-4551-b853-9beed051b54c\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " Oct 30 18:28:57.438702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.438637 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-openshift-ca\") pod \"e26cc1c5-716e-4551-b853-9beed051b54c\" (UID: \"e26cc1c5-716e-4551-b853-9beed051b54c\") " Oct 30 18:28:57.439016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.438989 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-openshift-ca" pod "e26cc1c5-716e-4551-b853-9beed051b54c" (UID: "e26cc1c5-716e-4551-b853-9beed051b54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:57.439210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.439195 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:57.440650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.440618 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26cc1c5-716e-4551-b853-9beed051b54c-kube-api-access-jj8z9" pod "e26cc1c5-716e-4551-b853-9beed051b54c" (UID: "e26cc1c5-716e-4551-b853-9beed051b54c"). InnerVolumeSpecName "kube-api-access-jj8z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:57.440650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.440619 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-xks-ca" pod "e26cc1c5-716e-4551-b853-9beed051b54c" (UID: "e26cc1c5-716e-4551-b853-9beed051b54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:57.540622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.540532 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj8z9\" (UniqueName: \"kubernetes.io/projected/e26cc1c5-716e-4551-b853-9beed051b54c-kube-api-access-jj8z9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:57.540622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.540573 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26cc1c5-716e-4551-b853-9beed051b54c-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:28:57.841454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:57.841370 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:58.214928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:58.214897 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" Oct 30 18:28:58.214928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:58.214923 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4" event={"ID":"e26cc1c5-716e-4551-b853-9beed051b54c","Type":"ContainerDied","Data":"2556b1584269b5be11a4c51cb13b35d4025d7094abf2f5ea912acdb0a766110f"} Oct 30 18:28:58.215145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:58.214972 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2556b1584269b5be11a4c51cb13b35d4025d7094abf2f5ea912acdb0a766110f" Oct 30 18:28:59.337178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:59.337145 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log" Oct 30 18:28:59.534194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:28:59.534169 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log" Oct 30 18:29:00.292682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.292644 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:00.515481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.515445 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8"] Oct 30 18:29:00.516053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.516037 4582 cpu_manager.go:414] "containerMap: removing container" podUID="e26cc1c5-716e-4551-b853-9beed051b54c" containerName="revoke-keys" Oct 30 18:29:00.516053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.516053 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26cc1c5-716e-4551-b853-9beed051b54c" containerName="revoke-keys" Oct 30 18:29:00.516163 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.516153 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26cc1c5-716e-4551-b853-9beed051b54c" containerName="revoke-keys" Oct 30 18:29:00.519549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.519533 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.533972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.533949 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:00.543409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.543348 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:29:00.547713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.547685 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8"] Oct 30 18:29:00.577074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.577043 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsfz\" (UniqueName: \"kubernetes.io/projected/bb75623c-6f3f-413d-b80c-bd341ddf25e0-kube-api-access-spsfz\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.577259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.577089 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.577259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.577105 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.678547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.678517 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsfz\" (UniqueName: \"kubernetes.io/projected/bb75623c-6f3f-413d-b80c-bd341ddf25e0-kube-api-access-spsfz\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.678683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.678564 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.678683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.678583 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.679177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.679159 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.680810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.680793 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.694555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.694538 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsfz\" (UniqueName: \"kubernetes.io/projected/bb75623c-6f3f-413d-b80c-bd341ddf25e0-kube-api-access-spsfz\") pod \"maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:00.833650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:00.833577 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:01.138885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:01.138851 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8"] Oct 30 18:29:01.143351 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:01.143317 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb75623c_6f3f_413d_b80c_bd341ddf25e0.slice/crio-9bc721ad341e8023134d02f95b31a21169a84a537b1e4a4466f3c093c737d78e WatchSource:0}: Error finding container 9bc721ad341e8023134d02f95b31a21169a84a537b1e4a4466f3c093c737d78e: Status 404 returned error can't find the container with id 9bc721ad341e8023134d02f95b31a21169a84a537b1e4a4466f3c093c737d78e Oct 30 18:29:01.235903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:01.235870 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" event={"ID":"bb75623c-6f3f-413d-b80c-bd341ddf25e0","Type":"ContainerStarted","Data":"9bc721ad341e8023134d02f95b31a21169a84a537b1e4a4466f3c093c737d78e"} Oct 30 18:29:02.160721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:02.160694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:02.243512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:02.243425 4582 generic.go:356] "Generic (PLEG): container finished" podID="bb75623c-6f3f-413d-b80c-bd341ddf25e0" containerID="d3110f73c8f79e2edb1a19fce4df41a1544dde099c0ee716723c0f492b8c56e9" exitCode=0 Oct 30 18:29:02.243680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:02.243511 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" event={"ID":"bb75623c-6f3f-413d-b80c-bd341ddf25e0","Type":"ContainerDied","Data":"d3110f73c8f79e2edb1a19fce4df41a1544dde099c0ee716723c0f492b8c56e9"} Oct 30 18:29:02.899715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:02.899685 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:03.412604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.412580 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:03.514736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.514668 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-openshift-ca\") pod \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " Oct 30 18:29:03.514904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.514788 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb75623c-6f3f-413d-b80c-bd341ddf25e0-kube-api-access-spsfz\" (UniqueName: \"kubernetes.io/projected/bb75623c-6f3f-413d-b80c-bd341ddf25e0-kube-api-access-spsfz\") pod \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " Oct 30 18:29:03.514904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.514865 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-xks-ca\") pod \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\" (UID: \"bb75623c-6f3f-413d-b80c-bd341ddf25e0\") " Oct 30 18:29:03.515088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.515060 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-openshift-ca" pod "bb75623c-6f3f-413d-b80c-bd341ddf25e0" (UID: "bb75623c-6f3f-413d-b80c-bd341ddf25e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:03.515242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.515227 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.516826 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.516799 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-xks-ca" pod "bb75623c-6f3f-413d-b80c-bd341ddf25e0" (UID: "bb75623c-6f3f-413d-b80c-bd341ddf25e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:03.516919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.516901 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb75623c-6f3f-413d-b80c-bd341ddf25e0-kube-api-access-spsfz" pod "bb75623c-6f3f-413d-b80c-bd341ddf25e0" (UID: "bb75623c-6f3f-413d-b80c-bd341ddf25e0"). InnerVolumeSpecName "kube-api-access-spsfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:03.615944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.615907 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spsfz\" (UniqueName: \"kubernetes.io/projected/bb75623c-6f3f-413d-b80c-bd341ddf25e0-kube-api-access-spsfz\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.615944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.615943 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75623c-6f3f-413d-b80c-bd341ddf25e0-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.981101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:03.981064 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:04.258636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:04.258552 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" Oct 30 18:29:04.258636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:04.258579 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8" event={"ID":"bb75623c-6f3f-413d-b80c-bd341ddf25e0","Type":"ContainerDied","Data":"9bc721ad341e8023134d02f95b31a21169a84a537b1e4a4466f3c093c737d78e"} Oct 30 18:29:04.258846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:04.258756 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc721ad341e8023134d02f95b31a21169a84a537b1e4a4466f3c093c737d78e" Oct 30 18:29:05.832927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.832897 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh"] Oct 30 18:29:05.833464 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.833450 4582 cpu_manager.go:414] "containerMap: removing container" podUID="bb75623c-6f3f-413d-b80c-bd341ddf25e0" containerName="revoke-keys" Oct 30 18:29:05.833527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.833464 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb75623c-6f3f-413d-b80c-bd341ddf25e0" containerName="revoke-keys" Oct 30 18:29:05.833582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.833553 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb75623c-6f3f-413d-b80c-bd341ddf25e0" containerName="revoke-keys" Oct 30 18:29:05.837104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.837085 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:05.850965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.850946 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:05.860326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.860304 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:29:05.864917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.864897 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh"] Oct 30 18:29:05.945643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.945614 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:05.945800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.945743 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqk2\" (UniqueName: \"kubernetes.io/projected/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-kube-api-access-fpqk2\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:05.945933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:05.945913 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.046953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.046922 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.047117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.047009 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqk2\" (UniqueName: \"kubernetes.io/projected/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-kube-api-access-fpqk2\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.047117 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.047061 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.047648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.047625 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.049248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.049224 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.063287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.063263 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqk2\" (UniqueName: \"kubernetes.io/projected/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-kube-api-access-fpqk2\") pod \"maas-api-revoke-sub-7bc09254-8k7gh\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.151321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.151289 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:06.234102 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:06.233988 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:29:06.351160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.351123 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:06.463853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.463746 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh"] Oct 30 18:29:06.467162 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:06.467133 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7689d0_a86d_49e2_8c4b_de9d3fe2192b.slice/crio-0056c0569c96df3301c3d6f387c3c837abfbb58a1dccaf4a2317a175edde6870 WatchSource:0}: Error finding container 0056c0569c96df3301c3d6f387c3c837abfbb58a1dccaf4a2317a175edde6870: Status 404 returned error can't find the container with id 0056c0569c96df3301c3d6f387c3c837abfbb58a1dccaf4a2317a175edde6870 Oct 30 18:29:06.674264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.674228 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29"] Oct 30 18:29:06.679580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.679558 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.695297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.695276 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29"] Oct 30 18:29:06.758988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.756447 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.758988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.756589 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.758988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.756697 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkx2\" (UniqueName: \"kubernetes.io/projected/81b6bb82-963d-4f3a-b2e9-fd8128354dec-kube-api-access-zdkx2\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.858249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.858205 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.858746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.858289 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.858746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.858344 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkx2\" (UniqueName: \"kubernetes.io/projected/81b6bb82-963d-4f3a-b2e9-fd8128354dec-kube-api-access-zdkx2\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.858983 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.858961 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.860619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.860596 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.875841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.875782 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkx2\" (UniqueName: \"kubernetes.io/projected/81b6bb82-963d-4f3a-b2e9-fd8128354dec-kube-api-access-zdkx2\") pod \"maas-api-revoke-sub-366f1a13-cgd29\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:06.992327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:06.992296 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:07.293140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:07.293096 4582 generic.go:356] "Generic (PLEG): container finished" podID="5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" containerID="60bbeabca6570c2baf4e3ad48d830a0d52f078eb9a72e42535babc1e0d391365" exitCode=0 Oct 30 18:29:07.302793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:07.302768 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" event={"ID":"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b","Type":"ContainerDied","Data":"60bbeabca6570c2baf4e3ad48d830a0d52f078eb9a72e42535babc1e0d391365"} Oct 30 18:29:07.302916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:07.302847 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29"] Oct 30 18:29:07.302916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:07.302880 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" event={"ID":"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b","Type":"ContainerStarted","Data":"0056c0569c96df3301c3d6f387c3c837abfbb58a1dccaf4a2317a175edde6870"} Oct 30 18:29:07.344717 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:07.344687 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b6bb82_963d_4f3a_b2e9_fd8128354dec.slice/crio-fc6f7ee8bfe709ce4bee86d9ea85ac87e4feef2265f32188ed0ff48bd3acebc4 WatchSource:0}: Error finding container fc6f7ee8bfe709ce4bee86d9ea85ac87e4feef2265f32188ed0ff48bd3acebc4: Status 404 returned error can't find the container with id fc6f7ee8bfe709ce4bee86d9ea85ac87e4feef2265f32188ed0ff48bd3acebc4 Oct 30 18:29:08.076050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.076008 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:08.301774 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.301743 4582 generic.go:356] "Generic (PLEG): container finished" podID="81b6bb82-963d-4f3a-b2e9-fd8128354dec" containerID="1f7ddb1cb24e5662ade39339d9da9104e47252b37a26f3148537887b20419c98" exitCode=0 Oct 30 18:29:08.301964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.301843 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" event={"ID":"81b6bb82-963d-4f3a-b2e9-fd8128354dec","Type":"ContainerDied","Data":"1f7ddb1cb24e5662ade39339d9da9104e47252b37a26f3148537887b20419c98"} Oct 30 18:29:08.301964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.301894 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" event={"ID":"81b6bb82-963d-4f3a-b2e9-fd8128354dec","Type":"ContainerStarted","Data":"fc6f7ee8bfe709ce4bee86d9ea85ac87e4feef2265f32188ed0ff48bd3acebc4"} Oct 30 18:29:08.450950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.450922 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:08.579035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.579001 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-xks-ca\") pod \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " Oct 30 18:29:08.579229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.579125 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-kube-api-access-fpqk2\" (UniqueName: \"kubernetes.io/projected/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-kube-api-access-fpqk2\") pod \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " Oct 30 18:29:08.579294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.579255 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-openshift-ca\") pod \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\" (UID: \"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b\") " Oct 30 18:29:08.579609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.579585 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-openshift-ca" pod "5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" (UID: "5c7689d0-a86d-49e2-8c4b-de9d3fe2192b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:08.579757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.579741 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.581169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.581149 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-xks-ca" pod "5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" (UID: "5c7689d0-a86d-49e2-8c4b-de9d3fe2192b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:08.581260 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.581180 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-kube-api-access-fpqk2" pod "5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" (UID: "5c7689d0-a86d-49e2-8c4b-de9d3fe2192b"). InnerVolumeSpecName "kube-api-access-fpqk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:08.680874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.680808 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.680874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:08.680867 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqk2\" (UniqueName: \"kubernetes.io/projected/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b-kube-api-access-fpqk2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:09.317207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.317181 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" Oct 30 18:29:09.317607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.317213 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh" event={"ID":"5c7689d0-a86d-49e2-8c4b-de9d3fe2192b","Type":"ContainerDied","Data":"0056c0569c96df3301c3d6f387c3c837abfbb58a1dccaf4a2317a175edde6870"} Oct 30 18:29:09.317607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.317257 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0056c0569c96df3301c3d6f387c3c837abfbb58a1dccaf4a2317a175edde6870" Oct 30 18:29:09.529616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.529593 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:09.607322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.606342 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b6bb82-963d-4f3a-b2e9-fd8128354dec-kube-api-access-zdkx2\" (UniqueName: \"kubernetes.io/projected/81b6bb82-963d-4f3a-b2e9-fd8128354dec-kube-api-access-zdkx2\") pod \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " Oct 30 18:29:09.607322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.606432 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-xks-ca\") pod \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " Oct 30 18:29:09.607322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.606596 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-openshift-ca\") pod \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\" (UID: \"81b6bb82-963d-4f3a-b2e9-fd8128354dec\") " Oct 30 18:29:09.607843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.607782 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-openshift-ca" pod "81b6bb82-963d-4f3a-b2e9-fd8128354dec" (UID: "81b6bb82-963d-4f3a-b2e9-fd8128354dec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:09.612243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.612216 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b6bb82-963d-4f3a-b2e9-fd8128354dec-kube-api-access-zdkx2" pod "81b6bb82-963d-4f3a-b2e9-fd8128354dec" (UID: "81b6bb82-963d-4f3a-b2e9-fd8128354dec"). InnerVolumeSpecName "kube-api-access-zdkx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:09.612394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.612227 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-xks-ca" pod "81b6bb82-963d-4f3a-b2e9-fd8128354dec" (UID: "81b6bb82-963d-4f3a-b2e9-fd8128354dec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:09.708783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.708748 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:09.708976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.708874 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81b6bb82-963d-4f3a-b2e9-fd8128354dec-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:09.708976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:09.708897 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkx2\" (UniqueName: \"kubernetes.io/projected/81b6bb82-963d-4f3a-b2e9-fd8128354dec-kube-api-access-zdkx2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:10.176536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:10.176502 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:10.325065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:10.325036 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" Oct 30 18:29:10.325428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:10.325062 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29" event={"ID":"81b6bb82-963d-4f3a-b2e9-fd8128354dec","Type":"ContainerDied","Data":"fc6f7ee8bfe709ce4bee86d9ea85ac87e4feef2265f32188ed0ff48bd3acebc4"} Oct 30 18:29:10.325428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:10.325110 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6f7ee8bfe709ce4bee86d9ea85ac87e4feef2265f32188ed0ff48bd3acebc4" Oct 30 18:29:10.691898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:10.691867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:10.990162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:10.990087 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:11.317344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:11.317262 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:11.586429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:11.586352 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:11.964062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:11.964030 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:13.758485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:13.758450 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:14.893457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:14.893422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:15.382606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.382574 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq"] Oct 30 18:29:15.383267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.383248 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" containerName="revoke-keys" Oct 30 18:29:15.383334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.383268 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" containerName="revoke-keys" Oct 30 18:29:15.383334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.383291 4582 cpu_manager.go:414] "containerMap: removing container" podUID="81b6bb82-963d-4f3a-b2e9-fd8128354dec" containerName="revoke-keys" Oct 30 18:29:15.383334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.383298 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b6bb82-963d-4f3a-b2e9-fd8128354dec" containerName="revoke-keys" Oct 30 18:29:15.383460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.383381 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" containerName="revoke-keys" Oct 30 18:29:15.383460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.383394 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b6bb82-963d-4f3a-b2e9-fd8128354dec" containerName="revoke-keys" Oct 30 18:29:15.388126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.388107 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.409856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.409806 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:29:15.410013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.409876 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:15.414209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.414184 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq"] Oct 30 18:29:15.475392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.475365 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4q7f\" (UniqueName: \"kubernetes.io/projected/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-kube-api-access-v4q7f\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.475529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.475421 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.475581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.475554 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.576482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.576454 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.576632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.576547 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4q7f\" (UniqueName: \"kubernetes.io/projected/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-kube-api-access-v4q7f\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.576632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.576588 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.577208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.577188 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.578748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.578726 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.593194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.593168 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4q7f\" (UniqueName: \"kubernetes.io/projected/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-kube-api-access-v4q7f\") pod \"maas-api-revoke-sub-08ea5e2b-cldnq\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:15.632666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.632573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:15.701749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:15.701720 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:16.004175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:16.004103 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq"] Oct 30 18:29:16.007643 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:16.007614 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292b9bf3_0be3_41e2_9c53_aeca7a904ee4.slice/crio-cf89fe994b3cfcb981b55b393a59d80cde7d1619d617a8604c16fa57c92266bc WatchSource:0}: Error finding container cf89fe994b3cfcb981b55b393a59d80cde7d1619d617a8604c16fa57c92266bc: Status 404 returned error can't find the container with id cf89fe994b3cfcb981b55b393a59d80cde7d1619d617a8604c16fa57c92266bc Oct 30 18:29:16.231511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:16.231474 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:16.366453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:16.366418 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" event={"ID":"292b9bf3-0be3-41e2-9c53-aeca7a904ee4","Type":"ContainerStarted","Data":"e290614f9c3638f448262108587457ffe7aa15a9505c4570a580d40540ad7342"} Oct 30 18:29:16.366611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:16.366472 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" event={"ID":"292b9bf3-0be3-41e2-9c53-aeca7a904ee4","Type":"ContainerStarted","Data":"cf89fe994b3cfcb981b55b393a59d80cde7d1619d617a8604c16fa57c92266bc"} Oct 30 18:29:16.399634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:16.399585 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" podStartSLOduration=1.399572502 podStartE2EDuration="1.399572502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:16.398127622 +0000 UTC m=+1853.222521913" watchObservedRunningTime="2026-10-30 18:29:16.399572502 +0000 UTC m=+1853.223966793" Oct 30 18:29:17.036545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.036463 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq"] Oct 30 18:29:17.040458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.040440 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.056811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.056786 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq"] Oct 30 18:29:17.169840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.169718 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.195978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.194582 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptgl\" (UniqueName: \"kubernetes.io/projected/540773a8-343a-4f34-afd4-88177b5999d4-kube-api-access-hptgl\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.195978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.194683 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.195978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.194787 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.296032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.295947 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptgl\" (UniqueName: \"kubernetes.io/projected/540773a8-343a-4f34-afd4-88177b5999d4-kube-api-access-hptgl\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.296192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.296052 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.296192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.296181 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.297058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.297038 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.297157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.297137 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.299596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.299573 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.315397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.315372 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptgl\" (UniqueName: \"kubernetes.io/projected/540773a8-343a-4f34-afd4-88177b5999d4-kube-api-access-hptgl\") pod \"maas-api-revoke-sub-fde4a6e0-f9bgq\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.353431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.353403 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:17.376458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.376428 4582 generic.go:356] "Generic (PLEG): container finished" podID="292b9bf3-0be3-41e2-9c53-aeca7a904ee4" containerID="e290614f9c3638f448262108587457ffe7aa15a9505c4570a580d40540ad7342" exitCode=0 Oct 30 18:29:17.376604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.376509 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" event={"ID":"292b9bf3-0be3-41e2-9c53-aeca7a904ee4","Type":"ContainerDied","Data":"e290614f9c3638f448262108587457ffe7aa15a9505c4570a580d40540ad7342"} Oct 30 18:29:17.522872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.522836 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf"] Oct 30 18:29:17.523104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.523080 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" containerID="cri-o://fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639" gracePeriod=30 Oct 30 18:29:17.549143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.549071 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.664882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:17.664843 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq"] Oct 30 18:29:17.693373 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:17.691804 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540773a8_343a_4f34_afd4_88177b5999d4.slice/crio-0607d53fe0da0bca227cfa3aa49743dd535caa53cdcd6f1077a062c2eab3c9d8 WatchSource:0}: Error finding container 0607d53fe0da0bca227cfa3aa49743dd535caa53cdcd6f1077a062c2eab3c9d8: Status 404 returned error can't find the container with id 0607d53fe0da0bca227cfa3aa49743dd535caa53cdcd6f1077a062c2eab3c9d8 Oct 30 18:29:18.104153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.104119 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:18.385177 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.385144 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" event={"ID":"540773a8-343a-4f34-afd4-88177b5999d4","Type":"ContainerStarted","Data":"0e467a90975b76b6726a388428edcfa439eeb9343ef5abf5c294daf5e37f1925"} Oct 30 18:29:18.385392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.385197 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" event={"ID":"540773a8-343a-4f34-afd4-88177b5999d4","Type":"ContainerStarted","Data":"0607d53fe0da0bca227cfa3aa49743dd535caa53cdcd6f1077a062c2eab3c9d8"} Oct 30 18:29:18.423454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.423406 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" podStartSLOduration=1.423392902 podStartE2EDuration="1.423392902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:18.420018036 +0000 UTC m=+1855.244412329" watchObservedRunningTime="2026-10-30 18:29:18.423392902 +0000 UTC m=+1855.247787194" Oct 30 18:29:18.537805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.537780 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:18.612738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.612702 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-xks-ca\") pod \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " Oct 30 18:29:18.612938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.612915 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-openshift-ca\") pod \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " Oct 30 18:29:18.613027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.613008 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-kube-api-access-v4q7f\" (UniqueName: \"kubernetes.io/projected/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-kube-api-access-v4q7f\") pod \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\" (UID: \"292b9bf3-0be3-41e2-9c53-aeca7a904ee4\") " Oct 30 18:29:18.613318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.613286 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-openshift-ca" pod "292b9bf3-0be3-41e2-9c53-aeca7a904ee4" (UID: "292b9bf3-0be3-41e2-9c53-aeca7a904ee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:18.613527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.613505 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:18.614998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.614967 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-xks-ca" pod "292b9bf3-0be3-41e2-9c53-aeca7a904ee4" (UID: "292b9bf3-0be3-41e2-9c53-aeca7a904ee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:18.615150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.615128 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-kube-api-access-v4q7f" pod "292b9bf3-0be3-41e2-9c53-aeca7a904ee4" (UID: "292b9bf3-0be3-41e2-9c53-aeca7a904ee4"). InnerVolumeSpecName "kube-api-access-v4q7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:18.714613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.714541 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:18.714613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:18.714571 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4q7f\" (UniqueName: \"kubernetes.io/projected/292b9bf3-0be3-41e2-9c53-aeca7a904ee4-kube-api-access-v4q7f\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.398677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:19.398637 4582 generic.go:356] "Generic (PLEG): container finished" podID="540773a8-343a-4f34-afd4-88177b5999d4" containerID="0e467a90975b76b6726a388428edcfa439eeb9343ef5abf5c294daf5e37f1925" exitCode=0 Oct 30 18:29:19.400236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:19.398720 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" event={"ID":"540773a8-343a-4f34-afd4-88177b5999d4","Type":"ContainerDied","Data":"0e467a90975b76b6726a388428edcfa439eeb9343ef5abf5c294daf5e37f1925"} Oct 30 18:29:19.401442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:19.400709 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" Oct 30 18:29:19.401442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:19.400760 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq" event={"ID":"292b9bf3-0be3-41e2-9c53-aeca7a904ee4","Type":"ContainerDied","Data":"cf89fe994b3cfcb981b55b393a59d80cde7d1619d617a8604c16fa57c92266bc"} Oct 30 18:29:19.401442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:19.400806 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf89fe994b3cfcb981b55b393a59d80cde7d1619d617a8604c16fa57c92266bc" Oct 30 18:29:19.965868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:19.965762 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:20.587951 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.587887 4582 patch_prober.go:29] interesting pod/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:29:20.600936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.598992 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:29:20.768806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.767899 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:20.947445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.947391 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-xks-ca\") pod \"540773a8-343a-4f34-afd4-88177b5999d4\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " Oct 30 18:29:20.947645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.947541 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540773a8-343a-4f34-afd4-88177b5999d4-kube-api-access-hptgl\" (UniqueName: \"kubernetes.io/projected/540773a8-343a-4f34-afd4-88177b5999d4-kube-api-access-hptgl\") pod \"540773a8-343a-4f34-afd4-88177b5999d4\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " Oct 30 18:29:20.947645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.947609 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-openshift-ca\") pod \"540773a8-343a-4f34-afd4-88177b5999d4\" (UID: \"540773a8-343a-4f34-afd4-88177b5999d4\") " Oct 30 18:29:20.948249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.948214 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-openshift-ca" pod "540773a8-343a-4f34-afd4-88177b5999d4" (UID: "540773a8-343a-4f34-afd4-88177b5999d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:20.950524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.950494 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-xks-ca" pod "540773a8-343a-4f34-afd4-88177b5999d4" (UID: "540773a8-343a-4f34-afd4-88177b5999d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:20.951332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:20.951304 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540773a8-343a-4f34-afd4-88177b5999d4-kube-api-access-hptgl" pod "540773a8-343a-4f34-afd4-88177b5999d4" (UID: "540773a8-343a-4f34-afd4-88177b5999d4"). InnerVolumeSpecName "kube-api-access-hptgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:21.049375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.049284 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:21.049375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.049320 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hptgl\" (UniqueName: \"kubernetes.io/projected/540773a8-343a-4f34-afd4-88177b5999d4-kube-api-access-hptgl\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:21.049375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.049337 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540773a8-343a-4f34-afd4-88177b5999d4-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:21.427734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.427699 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" event={"ID":"540773a8-343a-4f34-afd4-88177b5999d4","Type":"ContainerDied","Data":"0607d53fe0da0bca227cfa3aa49743dd535caa53cdcd6f1077a062c2eab3c9d8"} Oct 30 18:29:21.427945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.427737 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq" Oct 30 18:29:21.427945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.427749 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0607d53fe0da0bca227cfa3aa49743dd535caa53cdcd6f1077a062c2eab3c9d8" Oct 30 18:29:21.756893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.756842 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:21.882629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.882595 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb"] Oct 30 18:29:21.882801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.882775 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" podUID="df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" containerName="payload-processing" containerID="cri-o://479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f" gracePeriod=30 Oct 30 18:29:21.910221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.910188 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x"] Oct 30 18:29:21.910470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.910436 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" podUID="8bdd259e-80c6-4cea-855a-50e7f4ae4a66" containerName="payload-pre-processing" containerID="cri-o://0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4" gracePeriod=30 Oct 30 18:29:21.959687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:21.959654 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.062630 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.062591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.294091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.294013 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:29:22.297463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.297434 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:29:22.371582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.371544 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-plugins-config-volume\") pod \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " Oct 30 18:29:22.371803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.371721 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-kube-api-access-6sbhk\" (UniqueName: \"kubernetes.io/projected/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-kube-api-access-6sbhk\") pod \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " Oct 30 18:29:22.371931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.371878 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-plugins-config-volume\") pod \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\" (UID: \"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660\") " Oct 30 18:29:22.372012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.371942 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-plugins-config-volume" pod "8bdd259e-80c6-4cea-855a-50e7f4ae4a66" (UID: "8bdd259e-80c6-4cea-855a-50e7f4ae4a66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:22.372012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.371947 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-kube-api-access-xmjlk\" (UniqueName: \"kubernetes.io/projected/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-kube-api-access-xmjlk\") pod \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\" (UID: \"8bdd259e-80c6-4cea-855a-50e7f4ae4a66\") " Oct 30 18:29:22.372461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.372432 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-plugins-config-volume" pod "df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" (UID: "df92986b-c01f-4f9d-8ac1-ce6d5eaf7660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:22.375199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.375175 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-kube-api-access-6sbhk" pod "df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" (UID: "df92986b-c01f-4f9d-8ac1-ce6d5eaf7660"). InnerVolumeSpecName "kube-api-access-6sbhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:22.375415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.375399 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.375502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.375421 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.376853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.376810 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-kube-api-access-xmjlk" pod "8bdd259e-80c6-4cea-855a-50e7f4ae4a66" (UID: "8bdd259e-80c6-4cea-855a-50e7f4ae4a66"). InnerVolumeSpecName "kube-api-access-xmjlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:22.402382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.402356 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.436011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.435979 4582 generic.go:356] "Generic (PLEG): container finished" podID="df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" containerID="479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f" exitCode=0 Oct 30 18:29:22.436183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.436027 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" Oct 30 18:29:22.436183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.436062 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" event={"ID":"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660","Type":"ContainerDied","Data":"479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f"} Oct 30 18:29:22.436183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.436118 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb" event={"ID":"df92986b-c01f-4f9d-8ac1-ce6d5eaf7660","Type":"ContainerDied","Data":"088a7a5f8d80d00d423ae74c64d6ad453fa886a6648d4b0f1922a33f373fab42"} Oct 30 18:29:22.436183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.436146 4582 scope.go:122] "RemoveContainer" containerID="479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f" Oct 30 18:29:22.437848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.437811 4582 generic.go:356] "Generic (PLEG): container finished" podID="8bdd259e-80c6-4cea-855a-50e7f4ae4a66" containerID="0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4" exitCode=0 Oct 30 18:29:22.437958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.437875 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" Oct 30 18:29:22.437958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.437909 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" event={"ID":"8bdd259e-80c6-4cea-855a-50e7f4ae4a66","Type":"ContainerDied","Data":"0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4"} Oct 30 18:29:22.437958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.437954 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x" event={"ID":"8bdd259e-80c6-4cea-855a-50e7f4ae4a66","Type":"ContainerDied","Data":"70e436a35a9706abbb3065093ac706320108281f36ffdd30d6ac10377ff403c8"} Oct 30 18:29:22.451047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.451025 4582 scope.go:122] "RemoveContainer" containerID="479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f" Oct 30 18:29:22.451367 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:22.451336 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f\": container with ID starting with 479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f not found: ID does not exist" containerID="479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f" Oct 30 18:29:22.451469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.451378 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f"} err="failed to get container status \"479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f\": rpc error: code = NotFound desc = could not find container \"479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f\": container with ID starting with 479728b5483400277dac58a9b225e5d2fc00b8d72a046c5bb655129188c3778f not found: ID does not exist" Oct 30 18:29:22.451469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.451401 4582 scope.go:122] "RemoveContainer" containerID="0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4" Oct 30 18:29:22.460518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.460501 4582 scope.go:122] "RemoveContainer" containerID="0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4" Oct 30 18:29:22.460785 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:22.460768 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4\": container with ID starting with 0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4 not found: ID does not exist" containerID="0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4" Oct 30 18:29:22.460867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.460792 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4"} err="failed to get container status \"0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4\": rpc error: code = NotFound desc = could not find container \"0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4\": container with ID starting with 0e04f8e44cff95bd1dc8234489d013ba43bdf07a8a081878a13092f884fc85c4 not found: ID does not exist" Oct 30 18:29:22.476879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.476860 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbhk\" (UniqueName: \"kubernetes.io/projected/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660-kube-api-access-6sbhk\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.476970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.476884 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjlk\" (UniqueName: \"kubernetes.io/projected/8bdd259e-80c6-4cea-855a-50e7f4ae4a66-kube-api-access-xmjlk\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.498998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.498974 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb"] Oct 30 18:29:22.506379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.506355 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e9b146-74dbdb8b44-hzqmb"] Oct 30 18:29:22.536580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.536552 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.536712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.536694 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x"] Oct 30 18:29:22.540999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.540979 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e9b146-5bd74f8987-gcz4x"] Oct 30 18:29:22.583573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.583518 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8"] Oct 30 18:29:22.587048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.587028 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e9b146-cmwz8"] Oct 30 18:29:22.643151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.643123 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.769114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.769091 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:29:22.780221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780199 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-data\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780246 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-socket\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780340 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-kube-api-access-h6lhj\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-kube-api-access-h6lhj\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780393 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-credential-socket\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780449 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istiod-ca-cert\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780474 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-socket" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:22.780577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780481 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-data" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:22.780577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780497 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-token\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780570 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-podinfo\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780620 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-envoy\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780658 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-certs\") pod \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\" (UID: \"8d8534d1-561c-4aa7-8380-8bf63ad459e7\") " Oct 30 18:29:22.780793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780712 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-credential-socket" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:22.781079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.780960 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istiod-ca-cert" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:22.781342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.781305 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.781342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.781329 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.781453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.781344 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.781453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.781359 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.781608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.781586 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-certs" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:22.782716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.782693 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-token" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:22.782842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.782765 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-kube-api-access-h6lhj" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "kube-api-access-h6lhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:22.783071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.783054 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-podinfo" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:22.783071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.783062 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-envoy" pod "8d8534d1-561c-4aa7-8380-8bf63ad459e7" (UID: "8d8534d1-561c-4aa7-8380-8bf63ad459e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:22.882742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.882716 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.882742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.882740 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.882955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.882750 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8534d1-561c-4aa7-8380-8bf63ad459e7-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.882955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.882759 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6lhj\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-kube-api-access-h6lhj\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:22.882955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:22.882770 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8534d1-561c-4aa7-8380-8bf63ad459e7-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:23.047452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.047415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:23.243607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.243519 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2"] Oct 30 18:29:23.244143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244127 4582 cpu_manager.go:414] "containerMap: removing container" podUID="df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" containerName="payload-processing" Oct 30 18:29:23.244220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244144 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" containerName="payload-processing" Oct 30 18:29:23.244220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244168 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" Oct 30 18:29:23.244220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244175 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" Oct 30 18:29:23.244220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244200 4582 cpu_manager.go:414] "containerMap: removing container" podUID="540773a8-343a-4f34-afd4-88177b5999d4" containerName="revoke-keys" Oct 30 18:29:23.244220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244204 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540773a8-343a-4f34-afd4-88177b5999d4" containerName="revoke-keys" Oct 30 18:29:23.244220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244212 4582 cpu_manager.go:414] "containerMap: removing container" podUID="292b9bf3-0be3-41e2-9c53-aeca7a904ee4" containerName="revoke-keys" Oct 30 18:29:23.244220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244216 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292b9bf3-0be3-41e2-9c53-aeca7a904ee4" containerName="revoke-keys" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244226 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8bdd259e-80c6-4cea-855a-50e7f4ae4a66" containerName="payload-pre-processing" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244230 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdd259e-80c6-4cea-855a-50e7f4ae4a66" containerName="payload-pre-processing" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244239 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244244 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244343 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244355 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="540773a8-343a-4f34-afd4-88177b5999d4" containerName="revoke-keys" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244360 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="292b9bf3-0be3-41e2-9c53-aeca7a904ee4" containerName="revoke-keys" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244368 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdd259e-80c6-4cea-855a-50e7f4ae4a66" containerName="payload-pre-processing" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244374 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerName="istio-proxy" Oct 30 18:29:23.244430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.244383 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" containerName="payload-processing" Oct 30 18:29:23.248596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.248577 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.277233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.277203 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2"] Oct 30 18:29:23.287136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287092 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xpr\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-kube-api-access-46xpr\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287158 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287195 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287253 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287282 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287351 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287406 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287437 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.287561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.287475 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.304362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.304333 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdd259e-80c6-4cea-855a-50e7f4ae4a66" path="/var/lib/kubelet/pods/8bdd259e-80c6-4cea-855a-50e7f4ae4a66/volumes" Oct 30 18:29:23.304798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.304784 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb75623c-6f3f-413d-b80c-bd341ddf25e0" path="/var/lib/kubelet/pods/bb75623c-6f3f-413d-b80c-bd341ddf25e0/volumes" Oct 30 18:29:23.305226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.305212 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df92986b-c01f-4f9d-8ac1-ce6d5eaf7660" path="/var/lib/kubelet/pods/df92986b-c01f-4f9d-8ac1-ce6d5eaf7660/volumes" Oct 30 18:29:23.389080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389051 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.389249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389096 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xpr\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-kube-api-access-46xpr\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.389249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389133 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.389249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389163 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.389447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389230 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.389501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389480 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.389769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389742 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.389769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389713 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.390025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.389873 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.390025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.390001 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.390025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.390008 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.390208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.390060 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.390273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.390235 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.390607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.390591 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.391910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.391870 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.392022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.391988 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.407504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.407478 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.407999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.407980 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xpr\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-kube-api-access-46xpr\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.447546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.447517 4582 generic.go:356] "Generic (PLEG): container finished" podID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" containerID="fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639" exitCode=0 Oct 30 18:29:23.447667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.447562 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" Oct 30 18:29:23.447667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.447600 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" event={"ID":"8d8534d1-561c-4aa7-8380-8bf63ad459e7","Type":"ContainerDied","Data":"fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639"} Oct 30 18:29:23.447667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.447652 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf" event={"ID":"8d8534d1-561c-4aa7-8380-8bf63ad459e7","Type":"ContainerDied","Data":"ee0e3e3157208ebdceba1c29851f07104f03827ff116e8137167e68d922341f2"} Oct 30 18:29:23.447812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.447681 4582 scope.go:122] "RemoveContainer" containerID="fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639" Oct 30 18:29:23.457800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.457776 4582 scope.go:122] "RemoveContainer" containerID="54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8" Oct 30 18:29:23.466636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.466619 4582 scope.go:122] "RemoveContainer" containerID="fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639" Oct 30 18:29:23.466944 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:23.466927 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639\": container with ID starting with fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639 not found: ID does not exist" containerID="fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639" Oct 30 18:29:23.467008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.466955 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639"} err="failed to get container status \"fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639\": rpc error: code = NotFound desc = could not find container \"fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639\": container with ID starting with fdd0054ae5278b087311ab7874726365a63f8beb11d778fb216a0a2b31563639 not found: ID does not exist" Oct 30 18:29:23.467008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.466972 4582 scope.go:122] "RemoveContainer" containerID="54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8" Oct 30 18:29:23.467217 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:23.467198 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8\": container with ID starting with 54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8 not found: ID does not exist" containerID="54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8" Oct 30 18:29:23.467267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.467232 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8"} err="failed to get container status \"54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8\": rpc error: code = NotFound desc = could not find container \"54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8\": container with ID starting with 54b1fdf17f130e81d9171a22c4505cf12a998c5668a75cc665c7a4a266df78f8 not found: ID does not exist" Oct 30 18:29:23.492391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.492363 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf"] Oct 30 18:29:23.496549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.496502 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a4f3f0-openshift-default-668bf869d5-sjncf"] Oct 30 18:29:23.563557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.563531 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:23.877131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.877102 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2"] Oct 30 18:29:23.880270 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:23.880239 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad432de_0709_43a2_8eb1_cc5abd5c300a.slice/crio-0f3c96ffaf48dcd0af92df0dc4e2acdb120e4efa63d6854c25a542c06fb6d11f WatchSource:0}: Error finding container 0f3c96ffaf48dcd0af92df0dc4e2acdb120e4efa63d6854c25a542c06fb6d11f: Status 404 returned error can't find the container with id 0f3c96ffaf48dcd0af92df0dc4e2acdb120e4efa63d6854c25a542c06fb6d11f Oct 30 18:29:23.882681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.882649 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:23.882742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.882716 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:23.882789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:23.882746 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:24.460120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:24.460082 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" event={"ID":"5ad432de-0709-43a2-8eb1-cc5abd5c300a","Type":"ContainerStarted","Data":"aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625"} Oct 30 18:29:24.460308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:24.460130 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" event={"ID":"5ad432de-0709-43a2-8eb1-cc5abd5c300a","Type":"ContainerStarted","Data":"0f3c96ffaf48dcd0af92df0dc4e2acdb120e4efa63d6854c25a542c06fb6d11f"} Oct 30 18:29:24.479321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:24.479292 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:24.516357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:24.516313 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podStartSLOduration=1.516296477 podStartE2EDuration="1.516296477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:24.50495791 +0000 UTC m=+1861.329352202" watchObservedRunningTime="2026-10-30 18:29:24.516296477 +0000 UTC m=+1861.340690768" Oct 30 18:29:24.564535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:24.564501 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:24.566341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:24.566315 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:29:24.566494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:24.566361 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 18:29:25.301079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:25.301041 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8534d1-561c-4aa7-8380-8bf63ad459e7" path="/var/lib/kubelet/pods/8d8534d1-561c-4aa7-8380-8bf63ad459e7/volumes" Oct 30 18:29:25.565099 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:25.565008 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:29:25.565354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:25.565326 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 18:29:26.359336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:26.359298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:26.567242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:26.567203 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:29:26.567422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:26.567269 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 18:29:27.568602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:27.565230 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:29:27.568602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:27.565291 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 18:29:28.106154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:28.106105 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:28.565180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:28.565141 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:29:28.566005 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:28.565973 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 18:29:29.390988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:29.390949 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:29.524008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:29.523975 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:29.566845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:29.565148 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:29:29.566845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:29.565219 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 18:29:29.679922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:29.679835 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5"] Oct 30 18:29:29.680325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:29.680276 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" containerID="cri-o://7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0" gracePeriod=30 Oct 30 18:29:29.705644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:29.705597 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:30.166844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.166091 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:30.337449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.337375 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h"] Oct 30 18:29:30.343249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.343230 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.360862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.360303 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h"] Oct 30 18:29:30.366060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.366031 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:30.366236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.366102 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:29:30.510260 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.510227 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.510639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.510365 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m49n\" (UniqueName: \"kubernetes.io/projected/51c4d29d-bc00-487d-9319-337e4fe7ec58-kube-api-access-8m49n\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.510639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.510483 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.564698 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.564660 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:29:30.564917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.564723 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 18:29:30.612208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.612130 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.612208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.612206 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m49n\" (UniqueName: \"kubernetes.io/projected/51c4d29d-bc00-487d-9319-337e4fe7ec58-kube-api-access-8m49n\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.612375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.612278 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.612931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.612903 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.614458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.614436 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.631712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.631686 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m49n\" (UniqueName: \"kubernetes.io/projected/51c4d29d-bc00-487d-9319-337e4fe7ec58-kube-api-access-8m49n\") pod \"maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.656678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.656653 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:30.707866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.707747 4582 patch_prober.go:29] interesting pod/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:29:30.707866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.707807 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:29:30.963453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:30.963423 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h"] Oct 30 18:29:30.967303 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:30.967275 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c4d29d_bc00_487d_9319_337e4fe7ec58.slice/crio-876e2633ab58e55d99914db3fdcfce19cd656afb78480644c792e598ea0d331e WatchSource:0}: Error finding container 876e2633ab58e55d99914db3fdcfce19cd656afb78480644c792e598ea0d331e: Status 404 returned error can't find the container with id 876e2633ab58e55d99914db3fdcfce19cd656afb78480644c792e598ea0d331e Oct 30 18:29:31.527060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:31.526972 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" event={"ID":"51c4d29d-bc00-487d-9319-337e4fe7ec58","Type":"ContainerStarted","Data":"883f8224df30895289801d3acd9aa28b2cb81a62b20a18edf3d46fbf565cee9e"} Oct 30 18:29:31.527060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:31.527020 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" event={"ID":"51c4d29d-bc00-487d-9319-337e4fe7ec58","Type":"ContainerStarted","Data":"876e2633ab58e55d99914db3fdcfce19cd656afb78480644c792e598ea0d331e"} Oct 30 18:29:31.560928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:31.560873 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" podStartSLOduration=1.560848706 podStartE2EDuration="1.560848706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:31.558682789 +0000 UTC m=+1868.383077080" watchObservedRunningTime="2026-10-30 18:29:31.560848706 +0000 UTC m=+1868.385243000" Oct 30 18:29:31.598413 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:31.598376 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:31.598873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:31.598853 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:31.599632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:31.599591 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:29:32.094685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:32.094653 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:32.535924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:32.535878 4582 generic.go:356] "Generic (PLEG): container finished" podID="51c4d29d-bc00-487d-9319-337e4fe7ec58" containerID="883f8224df30895289801d3acd9aa28b2cb81a62b20a18edf3d46fbf565cee9e" exitCode=0 Oct 30 18:29:32.536375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:32.535963 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" event={"ID":"51c4d29d-bc00-487d-9319-337e4fe7ec58","Type":"ContainerDied","Data":"883f8224df30895289801d3acd9aa28b2cb81a62b20a18edf3d46fbf565cee9e"} Oct 30 18:29:33.689779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.689754 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:33.862395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.862319 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c4d29d-bc00-487d-9319-337e4fe7ec58-kube-api-access-8m49n\" (UniqueName: \"kubernetes.io/projected/51c4d29d-bc00-487d-9319-337e4fe7ec58-kube-api-access-8m49n\") pod \"51c4d29d-bc00-487d-9319-337e4fe7ec58\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " Oct 30 18:29:33.862395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.862393 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-xks-ca\") pod \"51c4d29d-bc00-487d-9319-337e4fe7ec58\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " Oct 30 18:29:33.862590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.862557 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-openshift-ca\") pod \"51c4d29d-bc00-487d-9319-337e4fe7ec58\" (UID: \"51c4d29d-bc00-487d-9319-337e4fe7ec58\") " Oct 30 18:29:33.862991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.862961 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-openshift-ca" pod "51c4d29d-bc00-487d-9319-337e4fe7ec58" (UID: "51c4d29d-bc00-487d-9319-337e4fe7ec58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:33.863095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.863040 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:33.864535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.864506 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-xks-ca" pod "51c4d29d-bc00-487d-9319-337e4fe7ec58" (UID: "51c4d29d-bc00-487d-9319-337e4fe7ec58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:33.864636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.864557 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c4d29d-bc00-487d-9319-337e4fe7ec58-kube-api-access-8m49n" pod "51c4d29d-bc00-487d-9319-337e4fe7ec58" (UID: "51c4d29d-bc00-487d-9319-337e4fe7ec58"). InnerVolumeSpecName "kube-api-access-8m49n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:33.964594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.964562 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m49n\" (UniqueName: \"kubernetes.io/projected/51c4d29d-bc00-487d-9319-337e4fe7ec58-kube-api-access-8m49n\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:33.964594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:33.964590 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c4d29d-bc00-487d-9319-337e4fe7ec58-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:34.188646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.188608 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh"] Oct 30 18:29:34.189254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.189234 4582 cpu_manager.go:414] "containerMap: removing container" podUID="51c4d29d-bc00-487d-9319-337e4fe7ec58" containerName="revoke-keys" Oct 30 18:29:34.189373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.189255 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c4d29d-bc00-487d-9319-337e4fe7ec58" containerName="revoke-keys" Oct 30 18:29:34.189437 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.189410 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c4d29d-bc00-487d-9319-337e4fe7ec58" containerName="revoke-keys" Oct 30 18:29:34.192985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.192963 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.213667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.213637 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh"] Oct 30 18:29:34.369153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.369117 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.369325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.369162 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htpfl\" (UniqueName: \"kubernetes.io/projected/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-kube-api-access-htpfl\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.369325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.369221 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.470814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.470740 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.470814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.470792 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htpfl\" (UniqueName: \"kubernetes.io/projected/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-kube-api-access-htpfl\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.471068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.470860 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.471475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.471453 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.473166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.473141 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.488248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.488226 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htpfl\" (UniqueName: \"kubernetes.io/projected/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-kube-api-access-htpfl\") pod \"maas-api-revoke-sub-200da0b7-89fzh\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.514209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.514188 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:34.525299 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.525277 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:34.557522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.557496 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" Oct 30 18:29:34.557659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.557498 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h" event={"ID":"51c4d29d-bc00-487d-9319-337e4fe7ec58","Type":"ContainerDied","Data":"876e2633ab58e55d99914db3fdcfce19cd656afb78480644c792e598ea0d331e"} Oct 30 18:29:34.557659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.557644 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876e2633ab58e55d99914db3fdcfce19cd656afb78480644c792e598ea0d331e" Oct 30 18:29:34.816391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.816360 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh"] Oct 30 18:29:34.819151 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:34.819113 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca78c81e_4d94_41b8_a4e6_9a9dca65ae0d.slice/crio-a82bd23462bbe345dff104c1f847aec3e08b39251635f343d16fd005fec65122 WatchSource:0}: Error finding container a82bd23462bbe345dff104c1f847aec3e08b39251635f343d16fd005fec65122: Status 404 returned error can't find the container with id a82bd23462bbe345dff104c1f847aec3e08b39251635f343d16fd005fec65122 Oct 30 18:29:34.948224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.948196 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:29:34.979985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.979957 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-credential-socket\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980005 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-podinfo\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980073 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50651ef2-6ef6-41d3-ab69-893124fb312b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50651ef2-6ef6-41d3-ab69-893124fb312b-istiod-ca-cert\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980100 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-data\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980148 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-socket\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980180 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-kube-api-access-pzh4l\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-kube-api-access-pzh4l\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980233 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-credential-socket" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:34.980488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980393 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-envoy\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980431 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50651ef2-6ef6-41d3-ab69-893124fb312b-istiod-ca-cert" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:34.980488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980435 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-data" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:34.980488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980450 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-certs\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980443 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-socket" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:34.980488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980482 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-token\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-token\") pod \"50651ef2-6ef6-41d3-ab69-893124fb312b\" (UID: \"50651ef2-6ef6-41d3-ab69-893124fb312b\") " Oct 30 18:29:34.980833 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.980764 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-certs" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:34.981037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.981019 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50651ef2-6ef6-41d3-ab69-893124fb312b-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:34.981104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.981044 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:34.981104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.981058 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:34.981104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.981072 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:34.981104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.981087 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:34.982247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.982223 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-podinfo" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:34.982434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.982418 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-token" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:34.982734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.982709 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-kube-api-access-pzh4l" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "kube-api-access-pzh4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:34.982809 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:34.982713 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-envoy" pod "50651ef2-6ef6-41d3-ab69-893124fb312b" (UID: "50651ef2-6ef6-41d3-ab69-893124fb312b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:35.082321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.082293 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:35.082321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.082318 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:35.082500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.082328 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50651ef2-6ef6-41d3-ab69-893124fb312b-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:35.082500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.082336 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzh4l\" (UniqueName: \"kubernetes.io/projected/50651ef2-6ef6-41d3-ab69-893124fb312b-kube-api-access-pzh4l\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:35.565687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.565597 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" event={"ID":"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d","Type":"ContainerStarted","Data":"742a1bed9ffc9efffe5dbaf83fb742d263b2978017b71707f75746e654fd3c94"} Oct 30 18:29:35.565687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.565659 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" event={"ID":"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d","Type":"ContainerStarted","Data":"a82bd23462bbe345dff104c1f847aec3e08b39251635f343d16fd005fec65122"} Oct 30 18:29:35.567341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.567312 4582 generic.go:356] "Generic (PLEG): container finished" podID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerID="7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0" exitCode=0 Oct 30 18:29:35.567445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.567352 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" Oct 30 18:29:35.567445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.567388 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" event={"ID":"50651ef2-6ef6-41d3-ab69-893124fb312b","Type":"ContainerDied","Data":"7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0"} Oct 30 18:29:35.567445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.567435 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5" event={"ID":"50651ef2-6ef6-41d3-ab69-893124fb312b","Type":"ContainerDied","Data":"9d697d6cedb92e29bbea734d0309886072e4472d8937fde61535422c53308ba0"} Oct 30 18:29:35.567553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.567455 4582 scope.go:122] "RemoveContainer" containerID="7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0" Oct 30 18:29:35.578101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.578085 4582 scope.go:122] "RemoveContainer" containerID="8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893" Oct 30 18:29:35.596329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.596312 4582 scope.go:122] "RemoveContainer" containerID="7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0" Oct 30 18:29:35.596569 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:35.596552 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0\": container with ID starting with 7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0 not found: ID does not exist" containerID="7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0" Oct 30 18:29:35.596618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.596576 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0"} err="failed to get container status \"7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0\": rpc error: code = NotFound desc = could not find container \"7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0\": container with ID starting with 7fa44b7fd448a054a89d00d3f2a08b4a47876b1f1f1a865ae5f3c8b387483fd0 not found: ID does not exist" Oct 30 18:29:35.596618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.596594 4582 scope.go:122] "RemoveContainer" containerID="8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893" Oct 30 18:29:35.596808 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:35.596789 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893\": container with ID starting with 8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893 not found: ID does not exist" containerID="8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893" Oct 30 18:29:35.596931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.596815 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893"} err="failed to get container status \"8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893\": rpc error: code = NotFound desc = could not find container \"8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893\": container with ID starting with 8574c0ddcff0ca3fa52a89c3c9ddc9a9db3f2d28ce029ac7bd1209f78e08b893 not found: ID does not exist" Oct 30 18:29:35.599143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.599110 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" podStartSLOduration=1.599099413 podStartE2EDuration="1.599099413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:35.598269059 +0000 UTC m=+1872.422663363" watchObservedRunningTime="2026-10-30 18:29:35.599099413 +0000 UTC m=+1872.423493705" Oct 30 18:29:35.630852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.630833 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5"] Oct 30 18:29:35.634233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.634209 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e9b146-openshift-default-5cfcd94dbf-fwbc5"] Oct 30 18:29:35.863946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:35.863806 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:36.224315 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:36.224281 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:29:36.271374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:36.271351 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:36.471026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:36.470990 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:36.575850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:36.575761 4582 generic.go:356] "Generic (PLEG): container finished" podID="ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" containerID="742a1bed9ffc9efffe5dbaf83fb742d263b2978017b71707f75746e654fd3c94" exitCode=0 Oct 30 18:29:36.576028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:36.575844 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" event={"ID":"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d","Type":"ContainerDied","Data":"742a1bed9ffc9efffe5dbaf83fb742d263b2978017b71707f75746e654fd3c94"} Oct 30 18:29:37.299992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.299958 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" path="/var/lib/kubelet/pods/50651ef2-6ef6-41d3-ab69-893124fb312b/volumes" Oct 30 18:29:37.735894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.735870 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:37.815035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.815002 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-xks-ca\") pod \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " Oct 30 18:29:37.815226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.815126 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-openshift-ca\") pod \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " Oct 30 18:29:37.815226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.815190 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-kube-api-access-htpfl\" (UniqueName: \"kubernetes.io/projected/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-kube-api-access-htpfl\") pod \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\" (UID: \"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d\") " Oct 30 18:29:37.815548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.815518 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-openshift-ca" pod "ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" (UID: "ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:37.817251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.817226 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-xks-ca" pod "ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" (UID: "ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:37.817344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.817295 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-kube-api-access-htpfl" pod "ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" (UID: "ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d"). InnerVolumeSpecName "kube-api-access-htpfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:37.917042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.917016 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:37.917042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.917041 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htpfl\" (UniqueName: \"kubernetes.io/projected/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-kube-api-access-htpfl\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:37.917220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.917053 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:37.983776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.983742 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5"] Oct 30 18:29:37.984315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984300 4582 cpu_manager.go:414] "containerMap: removing container" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" Oct 30 18:29:37.984368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984317 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" Oct 30 18:29:37.984368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984339 4582 cpu_manager.go:414] "containerMap: removing container" podUID="ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" containerName="revoke-keys" Oct 30 18:29:37.984368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984345 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" containerName="revoke-keys" Oct 30 18:29:37.984368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984359 4582 cpu_manager.go:414] "containerMap: removing container" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" Oct 30 18:29:37.984368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984364 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" Oct 30 18:29:37.984525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984465 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" containerName="revoke-keys" Oct 30 18:29:37.984525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.984474 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" Oct 30 18:29:37.988241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:37.988219 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.005484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.005453 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5"] Oct 30 18:29:38.018418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.018390 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwm5\" (UniqueName: \"kubernetes.io/projected/f992480b-5bf3-4ad3-9f54-dd0839b3564e-kube-api-access-7cwm5\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.018588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.018563 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.018668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.018615 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.120173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.120142 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.120362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.120196 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.120362 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.120349 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwm5\" (UniqueName: \"kubernetes.io/projected/f992480b-5bf3-4ad3-9f54-dd0839b3564e-kube-api-access-7cwm5\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.121006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.120982 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.122605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.122580 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.136697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.136673 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwm5\" (UniqueName: \"kubernetes.io/projected/f992480b-5bf3-4ad3-9f54-dd0839b3564e-kube-api-access-7cwm5\") pod \"maas-api-revoke-sub-228d5d05-2qzg5\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.225667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.225597 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp"] Oct 30 18:29:38.226269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.226254 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="50651ef2-6ef6-41d3-ab69-893124fb312b" containerName="istio-proxy" Oct 30 18:29:38.230861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.230842 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.234535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.234507 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:38.253119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.253096 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp"] Oct 30 18:29:38.302250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.302217 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:38.322345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322315 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bql\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-kube-api-access-k8bql\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.322468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322359 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.322524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322469 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.322576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322532 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.322720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322700 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.322875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322735 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.322875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322757 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.322875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322850 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ed87cc-00c3-47b9-b210-dc5a9a852487-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.323029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.322918 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.424497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424463 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.424709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424511 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.424709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424542 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.424709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424562 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.424709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424596 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ed87cc-00c3-47b9-b210-dc5a9a852487-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.424709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424676 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424859 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bql\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-kube-api-access-k8bql\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424907 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424932 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424970 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.424998 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.425102 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.425260 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.425541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.425515 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ed87cc-00c3-47b9-b210-dc5a9a852487-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.426974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.426949 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.427223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.427205 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.441027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.441003 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.441484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.441466 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bql\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-kube-api-access-k8bql\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.547317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.547241 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:38.591382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.591180 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" Oct 30 18:29:38.591382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.591230 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh" event={"ID":"ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d","Type":"ContainerDied","Data":"a82bd23462bbe345dff104c1f847aec3e08b39251635f343d16fd005fec65122"} Oct 30 18:29:38.591382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.591313 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a82bd23462bbe345dff104c1f847aec3e08b39251635f343d16fd005fec65122" Oct 30 18:29:38.604585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.604535 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5"] Oct 30 18:29:38.608719 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:38.608693 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf992480b_5bf3_4ad3_9f54_dd0839b3564e.slice/crio-085a4c954d02b28298afef7bab6675ad1b463df95a959ec72a73fafc437fcd50 WatchSource:0}: Error finding container 085a4c954d02b28298afef7bab6675ad1b463df95a959ec72a73fafc437fcd50: Status 404 returned error can't find the container with id 085a4c954d02b28298afef7bab6675ad1b463df95a959ec72a73fafc437fcd50 Oct 30 18:29:38.853062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.852992 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp"] Oct 30 18:29:38.860265 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:38.860232 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ed87cc_00c3_47b9_b210_dc5a9a852487.slice/crio-7ddf68c3a2eb8523cf24801cf5aecabddf0bdcebd4fd3b50ad6d0623252933d1 WatchSource:0}: Error finding container 7ddf68c3a2eb8523cf24801cf5aecabddf0bdcebd4fd3b50ad6d0623252933d1: Status 404 returned error can't find the container with id 7ddf68c3a2eb8523cf24801cf5aecabddf0bdcebd4fd3b50ad6d0623252933d1 Oct 30 18:29:38.863202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.863168 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:38.863275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.863235 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:38.863313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:38.863271 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:39.618989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:39.618946 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" event={"ID":"f992480b-5bf3-4ad3-9f54-dd0839b3564e","Type":"ContainerStarted","Data":"fb26283cb23c23e8d5d18de42104ccd9950a96c1770cb84a8952d05fd8db7a7f"} Oct 30 18:29:39.619559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:39.619532 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" event={"ID":"f992480b-5bf3-4ad3-9f54-dd0839b3564e","Type":"ContainerStarted","Data":"085a4c954d02b28298afef7bab6675ad1b463df95a959ec72a73fafc437fcd50"} Oct 30 18:29:39.635861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:39.631119 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" event={"ID":"11ed87cc-00c3-47b9-b210-dc5a9a852487","Type":"ContainerStarted","Data":"2db9dfb0d38d4a0979851857f4298c4307580e738a85a1d6766506b50aa60187"} Oct 30 18:29:39.635861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:39.631180 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" event={"ID":"11ed87cc-00c3-47b9-b210-dc5a9a852487","Type":"ContainerStarted","Data":"7ddf68c3a2eb8523cf24801cf5aecabddf0bdcebd4fd3b50ad6d0623252933d1"} Oct 30 18:29:39.663470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:39.663400 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" podStartSLOduration=2.663378973 podStartE2EDuration="2.663378973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:39.661298084 +0000 UTC m=+1876.485692405" watchObservedRunningTime="2026-10-30 18:29:39.663378973 +0000 UTC m=+1876.487773264" Oct 30 18:29:39.702296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:39.702245 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podStartSLOduration=1.702229337 podStartE2EDuration="1.702229337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:39.701124666 +0000 UTC m=+1876.525518969" watchObservedRunningTime="2026-10-30 18:29:39.702229337 +0000 UTC m=+1876.526623629" Oct 30 18:29:40.554844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:40.553923 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:40.563843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:40.557915 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:29:40.563843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:40.557968 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:29:40.659846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:40.659281 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:40.690847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:40.677689 4582 generic.go:356] "Generic (PLEG): container finished" podID="f992480b-5bf3-4ad3-9f54-dd0839b3564e" containerID="fb26283cb23c23e8d5d18de42104ccd9950a96c1770cb84a8952d05fd8db7a7f" exitCode=0 Oct 30 18:29:40.690847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:40.678934 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" event={"ID":"f992480b-5bf3-4ad3-9f54-dd0839b3564e","Type":"ContainerDied","Data":"fb26283cb23c23e8d5d18de42104ccd9950a96c1770cb84a8952d05fd8db7a7f"} Oct 30 18:29:41.561845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:41.557175 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:29:41.561845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:41.557251 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:29:41.896982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:41.896960 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:42.021600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.021566 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-openshift-ca\") pod \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " Oct 30 18:29:42.021801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.021637 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f992480b-5bf3-4ad3-9f54-dd0839b3564e-kube-api-access-7cwm5\" (UniqueName: \"kubernetes.io/projected/f992480b-5bf3-4ad3-9f54-dd0839b3564e-kube-api-access-7cwm5\") pod \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " Oct 30 18:29:42.021907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.021816 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-xks-ca\") pod \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\" (UID: \"f992480b-5bf3-4ad3-9f54-dd0839b3564e\") " Oct 30 18:29:42.021974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.021937 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-openshift-ca" pod "f992480b-5bf3-4ad3-9f54-dd0839b3564e" (UID: "f992480b-5bf3-4ad3-9f54-dd0839b3564e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:42.022288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.022270 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:42.023683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.023657 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-xks-ca" pod "f992480b-5bf3-4ad3-9f54-dd0839b3564e" (UID: "f992480b-5bf3-4ad3-9f54-dd0839b3564e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:42.023793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.023713 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f992480b-5bf3-4ad3-9f54-dd0839b3564e-kube-api-access-7cwm5" pod "f992480b-5bf3-4ad3-9f54-dd0839b3564e" (UID: "f992480b-5bf3-4ad3-9f54-dd0839b3564e"). InnerVolumeSpecName "kube-api-access-7cwm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:42.123268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.123193 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cwm5\" (UniqueName: \"kubernetes.io/projected/f992480b-5bf3-4ad3-9f54-dd0839b3564e-kube-api-access-7cwm5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:42.123268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.123220 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f992480b-5bf3-4ad3-9f54-dd0839b3564e-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:42.556900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.549235 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:29:42.556900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.549306 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:29:42.695160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.695126 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" event={"ID":"f992480b-5bf3-4ad3-9f54-dd0839b3564e","Type":"ContainerDied","Data":"085a4c954d02b28298afef7bab6675ad1b463df95a959ec72a73fafc437fcd50"} Oct 30 18:29:42.695353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.695165 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5" Oct 30 18:29:42.695353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:42.695173 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="085a4c954d02b28298afef7bab6675ad1b463df95a959ec72a73fafc437fcd50" Oct 30 18:29:43.085266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:43.085229 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:43.557847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:43.549270 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:29:43.557847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:43.549337 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:29:43.768267 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:43.768217 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:44.562223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:44.562189 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:29:44.562834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:44.562787 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:29:44.995457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:44.995415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:45.548221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:45.548179 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:29:45.548400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:45.548241 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:29:46.769388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:46.769350 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:47.139181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:47.139149 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:47.392626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:47.392530 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:47.392886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:47.392863 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:47.393737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:47.393719 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:29:47.820872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:47.820765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:48.206933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.206897 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j"] Oct 30 18:29:48.207198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.207171 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" podUID="e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" containerName="maas-api" containerID="cri-o://991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f" gracePeriod=30 Oct 30 18:29:48.263461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.263413 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:48.461070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.460955 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:48.490346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.490323 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:29:48.561196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.561160 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:48.621647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.621616 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-kube-api-access-v4lwm\" (UniqueName: \"kubernetes.io/projected/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-kube-api-access-v4lwm\") pod \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " Oct 30 18:29:48.621813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.621686 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-metrics-certs\") pod \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " Oct 30 18:29:48.621813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.621724 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-openshift-service-ca\") pod \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " Oct 30 18:29:48.621993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.621863 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls\") pod \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\" (UID: \"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2\") " Oct 30 18:29:48.622192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.622163 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-openshift-service-ca" pod "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" (UID: "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:48.623764 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.623743 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls" pod "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" (UID: "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:48.623885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.623800 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-metrics-certs" pod "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" (UID: "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:48.623885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.623846 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-kube-api-access-v4lwm" pod "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" (UID: "e941aa1e-9529-4b0c-b7b2-e70e63bb53b2"). InnerVolumeSpecName "kube-api-access-v4lwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:48.723282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.723207 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:48.723282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.723232 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:48.723282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.723242 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:48.723282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.723250 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lwm\" (UniqueName: \"kubernetes.io/projected/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2-kube-api-access-v4lwm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:48.748092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.748060 4582 generic.go:356] "Generic (PLEG): container finished" podID="e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" containerID="991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f" exitCode=0 Oct 30 18:29:48.748234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.748104 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" Oct 30 18:29:48.748234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.748155 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" event={"ID":"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2","Type":"ContainerDied","Data":"991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f"} Oct 30 18:29:48.748234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.748205 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j" event={"ID":"e941aa1e-9529-4b0c-b7b2-e70e63bb53b2","Type":"ContainerDied","Data":"e202f2be8de4063f67b38e1e0d0523e1bf3871662441bf3bfa3c176b661663d1"} Oct 30 18:29:48.748234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.748228 4582 scope.go:122] "RemoveContainer" containerID="991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f" Oct 30 18:29:48.759113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.759059 4582 scope.go:122] "RemoveContainer" containerID="991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f" Oct 30 18:29:48.759302 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:29:48.759281 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f\": container with ID starting with 991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f not found: ID does not exist" containerID="991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f" Oct 30 18:29:48.759359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.759310 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f"} err="failed to get container status \"991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f\": rpc error: code = NotFound desc = could not find container \"991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f\": container with ID starting with 991512b9ac8f70ddecc5679d6904d647b5763e14143ed253baff84d9b96fda0f not found: ID does not exist" Oct 30 18:29:48.878866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.871408 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j"] Oct 30 18:29:48.878866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.875008 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-44379a-7dc7555d4d-jkq7j"] Oct 30 18:29:48.895606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:48.895573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:49.052542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.052466 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:49.105595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.105566 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h"] Oct 30 18:29:49.109003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.108963 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-44379a-t8t5h"] Oct 30 18:29:49.139144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.139116 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:49.198999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.198968 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj"] Oct 30 18:29:49.199498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.199486 4582 cpu_manager.go:414] "containerMap: removing container" podUID="e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" containerName="maas-api" Oct 30 18:29:49.199547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.199500 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" containerName="maas-api" Oct 30 18:29:49.199547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.199514 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f992480b-5bf3-4ad3-9f54-dd0839b3564e" containerName="revoke-keys" Oct 30 18:29:49.199547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.199518 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f992480b-5bf3-4ad3-9f54-dd0839b3564e" containerName="revoke-keys" Oct 30 18:29:49.199643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.199631 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" containerName="maas-api" Oct 30 18:29:49.199643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.199641 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f992480b-5bf3-4ad3-9f54-dd0839b3564e" containerName="revoke-keys" Oct 30 18:29:49.204069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.204046 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.218409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.218379 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:49.218549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.218516 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:29:49.218666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.218650 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:29:49.218727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.218651 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:29:49.222063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.222045 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj"] Oct 30 18:29:49.235595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.235558 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.235791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.235618 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85zh\" (UniqueName: \"kubernetes.io/projected/abb20b4a-e59d-4cb5-a95b-2d008268dd98-kube-api-access-w85zh\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.235791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.235754 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.299634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.299604 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c4d29d-bc00-487d-9319-337e4fe7ec58" path="/var/lib/kubelet/pods/51c4d29d-bc00-487d-9319-337e4fe7ec58/volumes" Oct 30 18:29:49.300074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.300060 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e941aa1e-9529-4b0c-b7b2-e70e63bb53b2" path="/var/lib/kubelet/pods/e941aa1e-9529-4b0c-b7b2-e70e63bb53b2/volumes" Oct 30 18:29:49.336917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.336866 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.336917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.336900 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85zh\" (UniqueName: \"kubernetes.io/projected/abb20b4a-e59d-4cb5-a95b-2d008268dd98-kube-api-access-w85zh\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.337095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.336926 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.337525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.337502 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.339712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.339675 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.353528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.353505 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85zh\" (UniqueName: \"kubernetes.io/projected/abb20b4a-e59d-4cb5-a95b-2d008268dd98-kube-api-access-w85zh\") pod \"maas-api-revoke-sub-0a9505ee-nlfsj\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.518724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.518693 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:49.582763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.582727 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:49.823870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:49.823721 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj"] Oct 30 18:29:49.827223 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:49.827200 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb20b4a_e59d_4cb5_a95b_2d008268dd98.slice/crio-9352d4793f4d503a0a19369e27228dfb2236fbe2b32e504bc7a2e064a29c4c02 WatchSource:0}: Error finding container 9352d4793f4d503a0a19369e27228dfb2236fbe2b32e504bc7a2e064a29c4c02: Status 404 returned error can't find the container with id 9352d4793f4d503a0a19369e27228dfb2236fbe2b32e504bc7a2e064a29c4c02 Oct 30 18:29:50.765363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:50.765331 4582 generic.go:356] "Generic (PLEG): container finished" podID="abb20b4a-e59d-4cb5-a95b-2d008268dd98" containerID="95f0e04fdde12cba9da1655d8d10f8858fda6f3890e14c688ca463147bd8bdad" exitCode=0 Oct 30 18:29:50.765766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:50.765421 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" event={"ID":"abb20b4a-e59d-4cb5-a95b-2d008268dd98","Type":"ContainerDied","Data":"95f0e04fdde12cba9da1655d8d10f8858fda6f3890e14c688ca463147bd8bdad"} Oct 30 18:29:50.765766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:50.765488 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" event={"ID":"abb20b4a-e59d-4cb5-a95b-2d008268dd98","Type":"ContainerStarted","Data":"9352d4793f4d503a0a19369e27228dfb2236fbe2b32e504bc7a2e064a29c4c02"} Oct 30 18:29:51.069273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.069183 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:51.756428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.756390 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m"] Oct 30 18:29:51.761065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.761046 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.778552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.778526 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m"] Oct 30 18:29:51.867519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.867487 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.867669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.867615 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.867751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.867729 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgjz\" (UniqueName: \"kubernetes.io/projected/1472e70f-57eb-4748-9465-5a7333fec068-kube-api-access-dmgjz\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.930715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.930689 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:51.969251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.969218 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-xks-ca\") pod \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " Oct 30 18:29:51.969959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.969911 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb20b4a-e59d-4cb5-a95b-2d008268dd98-kube-api-access-w85zh\" (UniqueName: \"kubernetes.io/projected/abb20b4a-e59d-4cb5-a95b-2d008268dd98-kube-api-access-w85zh\") pod \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " Oct 30 18:29:51.970073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.969960 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-openshift-ca\") pod \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\" (UID: \"abb20b4a-e59d-4cb5-a95b-2d008268dd98\") " Oct 30 18:29:51.970288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.970247 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.970547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.970375 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.970547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.970480 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgjz\" (UniqueName: \"kubernetes.io/projected/1472e70f-57eb-4748-9465-5a7333fec068-kube-api-access-dmgjz\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.971442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.971414 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-openshift-ca" pod "abb20b4a-e59d-4cb5-a95b-2d008268dd98" (UID: "abb20b4a-e59d-4cb5-a95b-2d008268dd98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:51.971662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.971641 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.972878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.972634 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-xks-ca" pod "abb20b4a-e59d-4cb5-a95b-2d008268dd98" (UID: "abb20b4a-e59d-4cb5-a95b-2d008268dd98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:51.973515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.973490 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:51.973669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.973648 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb20b4a-e59d-4cb5-a95b-2d008268dd98-kube-api-access-w85zh" pod "abb20b4a-e59d-4cb5-a95b-2d008268dd98" (UID: "abb20b4a-e59d-4cb5-a95b-2d008268dd98"). InnerVolumeSpecName "kube-api-access-w85zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:51.987884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:51.987855 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgjz\" (UniqueName: \"kubernetes.io/projected/1472e70f-57eb-4748-9465-5a7333fec068-kube-api-access-dmgjz\") pod \"maas-api-revoke-sub-1cfd6d6b-qgg4m\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:52.071917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.071862 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w85zh\" (UniqueName: \"kubernetes.io/projected/abb20b4a-e59d-4cb5-a95b-2d008268dd98-kube-api-access-w85zh\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:52.071917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.071885 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:52.071917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.071895 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb20b4a-e59d-4cb5-a95b-2d008268dd98-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:52.075496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.075479 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:52.371102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.370891 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m"] Oct 30 18:29:52.374221 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:52.374156 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1472e70f_57eb_4748_9465_5a7333fec068.slice/crio-285bb5d2af329ccaa449d34a4c7e44d01a3e55e4d9622b01faf59f657d697546 WatchSource:0}: Error finding container 285bb5d2af329ccaa449d34a4c7e44d01a3e55e4d9622b01faf59f657d697546: Status 404 returned error can't find the container with id 285bb5d2af329ccaa449d34a4c7e44d01a3e55e4d9622b01faf59f657d697546 Oct 30 18:29:52.763540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.763512 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:52.782296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.782271 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" Oct 30 18:29:52.782747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.782303 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj" event={"ID":"abb20b4a-e59d-4cb5-a95b-2d008268dd98","Type":"ContainerDied","Data":"9352d4793f4d503a0a19369e27228dfb2236fbe2b32e504bc7a2e064a29c4c02"} Oct 30 18:29:52.782747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.782354 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9352d4793f4d503a0a19369e27228dfb2236fbe2b32e504bc7a2e064a29c4c02" Oct 30 18:29:52.783989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.783964 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" event={"ID":"1472e70f-57eb-4748-9465-5a7333fec068","Type":"ContainerStarted","Data":"db59485300c486498306cac60c8e3a3899bb48434fb1f4ce5a144833d895a77d"} Oct 30 18:29:52.784141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.784010 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" event={"ID":"1472e70f-57eb-4748-9465-5a7333fec068","Type":"ContainerStarted","Data":"285bb5d2af329ccaa449d34a4c7e44d01a3e55e4d9622b01faf59f657d697546"} Oct 30 18:29:52.829685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:52.822767 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" podStartSLOduration=1.8227029670000001 podStartE2EDuration="1.822702967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:52.8172238 +0000 UTC m=+1889.641618092" watchObservedRunningTime="2026-10-30 18:29:52.822702967 +0000 UTC m=+1889.647097252" Oct 30 18:29:53.235896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.235861 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp"] Oct 30 18:29:53.236597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.236577 4582 cpu_manager.go:414] "containerMap: removing container" podUID="abb20b4a-e59d-4cb5-a95b-2d008268dd98" containerName="revoke-keys" Oct 30 18:29:53.236646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.236600 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb20b4a-e59d-4cb5-a95b-2d008268dd98" containerName="revoke-keys" Oct 30 18:29:53.236721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.236713 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb20b4a-e59d-4cb5-a95b-2d008268dd98" containerName="revoke-keys" Oct 30 18:29:53.240123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.240096 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.264010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.263978 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp"] Oct 30 18:29:53.284269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284239 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284331 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284364 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284400 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284432 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284462 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4djj\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-kube-api-access-d4djj\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284572 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284656 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.284782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.284740 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.385679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385643 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.385877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385693 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.385877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385748 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.385877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385841 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.385877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385873 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385902 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385937 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.385968 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4djj\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-kube-api-access-d4djj\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.386073 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.386181 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.386224 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.386353 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.386442 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.386615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.386593 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.388193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.388166 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.388415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.388399 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.401775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.401747 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4djj\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-kube-api-access-d4djj\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.401898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.401808 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.556473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.556353 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:53.806329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.806297 4582 generic.go:356] "Generic (PLEG): container finished" podID="1472e70f-57eb-4748-9465-5a7333fec068" containerID="db59485300c486498306cac60c8e3a3899bb48434fb1f4ce5a144833d895a77d" exitCode=0 Oct 30 18:29:53.806758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.806371 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" event={"ID":"1472e70f-57eb-4748-9465-5a7333fec068","Type":"ContainerDied","Data":"db59485300c486498306cac60c8e3a3899bb48434fb1f4ce5a144833d895a77d"} Oct 30 18:29:53.878651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.878624 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp"] Oct 30 18:29:53.882185 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:53.882156 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e62f1ef_ed97_4426_aa3d_3e304f50d4d8.slice/crio-a39fe1cb0d3fca65ab8b3f53888f4d3847c99d65852c2b82547f6dcc756eb31e WatchSource:0}: Error finding container a39fe1cb0d3fca65ab8b3f53888f4d3847c99d65852c2b82547f6dcc756eb31e: Status 404 returned error can't find the container with id a39fe1cb0d3fca65ab8b3f53888f4d3847c99d65852c2b82547f6dcc756eb31e Oct 30 18:29:53.885022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.884992 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:53.885100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.885055 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:53.885100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:53.885083 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:29:54.666710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:54.666673 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:54.814615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:54.814581 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" event={"ID":"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8","Type":"ContainerStarted","Data":"f278688c1593cc97386baa3261f634adf7193786fe1c6a82fee2b2f82fe2cd28"} Oct 30 18:29:54.815111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:54.814637 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" event={"ID":"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8","Type":"ContainerStarted","Data":"a39fe1cb0d3fca65ab8b3f53888f4d3847c99d65852c2b82547f6dcc756eb31e"} Oct 30 18:29:54.863386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:54.863321 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podStartSLOduration=1.8632991429999999 podStartE2EDuration="1.863299143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:54.859702565 +0000 UTC m=+1891.684096858" watchObservedRunningTime="2026-10-30 18:29:54.863299143 +0000 UTC m=+1891.687693442" Oct 30 18:29:54.980178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:54.980151 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:55.006669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.006638 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1472e70f-57eb-4748-9465-5a7333fec068-kube-api-access-dmgjz\" (UniqueName: \"kubernetes.io/projected/1472e70f-57eb-4748-9465-5a7333fec068-kube-api-access-dmgjz\") pod \"1472e70f-57eb-4748-9465-5a7333fec068\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " Oct 30 18:29:55.006805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.006729 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-xks-ca\") pod \"1472e70f-57eb-4748-9465-5a7333fec068\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " Oct 30 18:29:55.006901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.006888 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-openshift-ca\") pod \"1472e70f-57eb-4748-9465-5a7333fec068\" (UID: \"1472e70f-57eb-4748-9465-5a7333fec068\") " Oct 30 18:29:55.007226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.007201 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-openshift-ca" pod "1472e70f-57eb-4748-9465-5a7333fec068" (UID: "1472e70f-57eb-4748-9465-5a7333fec068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:55.007376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.007361 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:55.008626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.008605 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1472e70f-57eb-4748-9465-5a7333fec068-kube-api-access-dmgjz" pod "1472e70f-57eb-4748-9465-5a7333fec068" (UID: "1472e70f-57eb-4748-9465-5a7333fec068"). InnerVolumeSpecName "kube-api-access-dmgjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:55.008728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.008669 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-xks-ca" pod "1472e70f-57eb-4748-9465-5a7333fec068" (UID: "1472e70f-57eb-4748-9465-5a7333fec068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:55.108470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.108422 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmgjz\" (UniqueName: \"kubernetes.io/projected/1472e70f-57eb-4748-9465-5a7333fec068-kube-api-access-dmgjz\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:55.108470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.108461 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472e70f-57eb-4748-9465-5a7333fec068-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:29:55.566231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.563212 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:29:55.566961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.566933 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:29:55.567101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.566991 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 18:29:55.680291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.680253 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.782773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.782585 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.823460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.823377 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" Oct 30 18:29:55.823460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.823420 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m" event={"ID":"1472e70f-57eb-4748-9465-5a7333fec068","Type":"ContainerDied","Data":"285bb5d2af329ccaa449d34a4c7e44d01a3e55e4d9622b01faf59f657d697546"} Oct 30 18:29:55.824040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:55.823475 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285bb5d2af329ccaa449d34a4c7e44d01a3e55e4d9622b01faf59f657d697546" Oct 30 18:29:56.015266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.015230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:56.463844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.461607 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9"] Oct 30 18:29:56.463844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.462476 4582 cpu_manager.go:414] "containerMap: removing container" podUID="1472e70f-57eb-4748-9465-5a7333fec068" containerName="revoke-keys" Oct 30 18:29:56.463844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.462496 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1472e70f-57eb-4748-9465-5a7333fec068" containerName="revoke-keys" Oct 30 18:29:56.463844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.462638 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="1472e70f-57eb-4748-9465-5a7333fec068" containerName="revoke-keys" Oct 30 18:29:56.471264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.471211 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.477137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.477112 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9"] Oct 30 18:29:56.493089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.493050 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:29:56.493265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.493249 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:29:56.506932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.506910 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:56.512161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.512144 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:29:56.524926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.524894 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.525069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.524958 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncz9\" (UniqueName: \"kubernetes.io/projected/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-kube-api-access-8ncz9\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.525120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.525088 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.539434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.539408 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:56.569249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.569216 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:29:56.569420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.569277 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 18:29:56.626990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.626951 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.627187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.627168 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.627265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.627217 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncz9\" (UniqueName: \"kubernetes.io/projected/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-kube-api-access-8ncz9\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.628143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.628098 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.630911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.630525 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.663751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.663718 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncz9\" (UniqueName: \"kubernetes.io/projected/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-kube-api-access-8ncz9\") pod \"maas-api-revoke-sub-dde2b9d4-zbbk9\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:56.787758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:56.787673 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:29:57.124047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:57.124020 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9"] Oct 30 18:29:57.126724 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:29:57.126695 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b1fbfd_2dfc_4617_81dc_9eb1ded2b795.slice/crio-2be8267a984301d11588c9ef753e4d774213f345299fbcb1c50438e4d2af2f1e WatchSource:0}: Error finding container 2be8267a984301d11588c9ef753e4d774213f345299fbcb1c50438e4d2af2f1e: Status 404 returned error can't find the container with id 2be8267a984301d11588c9ef753e4d774213f345299fbcb1c50438e4d2af2f1e Oct 30 18:29:57.566846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:57.563522 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:29:57.566846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:57.563654 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 18:29:57.839258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:57.839171 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" event={"ID":"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795","Type":"ContainerStarted","Data":"c86403726d4cd8b61ad5265cfe6231d9b288a3587235be993131044275561bbb"} Oct 30 18:29:57.839258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:57.839245 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" event={"ID":"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795","Type":"ContainerStarted","Data":"2be8267a984301d11588c9ef753e4d774213f345299fbcb1c50438e4d2af2f1e"} Oct 30 18:29:57.884013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:57.883963 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" podStartSLOduration=1.883942948 podStartE2EDuration="1.883942948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:57.882991352 +0000 UTC m=+1894.707385664" watchObservedRunningTime="2026-10-30 18:29:57.883942948 +0000 UTC m=+1894.708337239" Oct 30 18:29:58.511612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:58.511524 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:58.562845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:58.562697 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:29:58.562845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:58.562754 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 18:29:58.872945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:58.872904 4582 generic.go:356] "Generic (PLEG): container finished" podID="25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" containerID="c86403726d4cd8b61ad5265cfe6231d9b288a3587235be993131044275561bbb" exitCode=0 Oct 30 18:29:58.873123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:58.872947 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" event={"ID":"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795","Type":"ContainerDied","Data":"c86403726d4cd8b61ad5265cfe6231d9b288a3587235be993131044275561bbb"} Oct 30 18:29:59.561845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.558157 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:29:59.561845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.558218 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 18:29:59.574198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.574169 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75"] Oct 30 18:29:59.578414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.578399 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.595628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.595600 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75"] Oct 30 18:29:59.684142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.684107 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.684142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.684146 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.684428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.684391 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xvp\" (UniqueName: \"kubernetes.io/projected/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-kube-api-access-92xvp\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.785235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.785199 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xvp\" (UniqueName: \"kubernetes.io/projected/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-kube-api-access-92xvp\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.785404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.785330 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.785404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.785360 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.786035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.786010 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.787815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.787786 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.801215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.801187 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xvp\" (UniqueName: \"kubernetes.io/projected/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-kube-api-access-92xvp\") pod \"maas-api-revoke-sub-42535f1c-s7f75\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.895367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.894229 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:29:59.909671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.904919 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f"] Oct 30 18:29:59.912087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.911051 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:29:59.918347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.918317 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f"] Oct 30 18:29:59.996729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.996685 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:29:59.996889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.996851 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mfp\" (UniqueName: \"kubernetes.io/projected/f192d012-e8bc-496b-a03c-5cfb8820f38d-kube-api-access-p6mfp\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:29:59.997040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:29:59.997003 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.101515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.101001 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.101515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.101105 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mfp\" (UniqueName: \"kubernetes.io/projected/f192d012-e8bc-496b-a03c-5cfb8820f38d-kube-api-access-p6mfp\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.101515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.101204 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.104234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.102230 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.115292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.115266 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.124906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.124880 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mfp\" (UniqueName: \"kubernetes.io/projected/f192d012-e8bc-496b-a03c-5cfb8820f38d-kube-api-access-p6mfp\") pod \"maas-api-revoke-sub-ffbf2f7d-fpd9f\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.167310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.165973 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw"] Oct 30 18:30:00.191195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.191002 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:00.194191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.193525 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w"] Oct 30 18:30:00.206051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.203476 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7"] Oct 30 18:30:00.206051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.204025 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.210949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.210848 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw"] Oct 30 18:30:00.211107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.210958 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.294845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.285994 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w"] Oct 30 18:30:00.308170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.304870 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4927ea-a695-4eaa-8443-2e83a61db882-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.308170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.304949 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzt4\" (UniqueName: \"kubernetes.io/projected/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-kube-api-access-nzzt4\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:00.308170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.305029 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:00.308170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.305470 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpbr\" (UniqueName: \"kubernetes.io/projected/9c4927ea-a695-4eaa-8443-2e83a61db882-kube-api-access-hqpbr\") pod \"maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.308170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.305598 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc662d85-9037-48e9-b509-0c2ace6fb267-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.308170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.305697 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqrh\" (UniqueName: \"kubernetes.io/projected/cc662d85-9037-48e9-b509-0c2ace6fb267-kube-api-access-mmqrh\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.315944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.314811 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7"] Oct 30 18:30:00.339197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.329462 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:00.420570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.420470 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:00.424995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.424942 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpbr\" (UniqueName: \"kubernetes.io/projected/9c4927ea-a695-4eaa-8443-2e83a61db882-kube-api-access-hqpbr\") pod \"maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.425204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.425186 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc662d85-9037-48e9-b509-0c2ace6fb267-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.425475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.425432 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqrh\" (UniqueName: \"kubernetes.io/projected/cc662d85-9037-48e9-b509-0c2ace6fb267-kube-api-access-mmqrh\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.425884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.425865 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4927ea-a695-4eaa-8443-2e83a61db882-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.426039 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.426025 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzt4\" (UniqueName: \"kubernetes.io/projected/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-kube-api-access-nzzt4\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:00.427102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.427078 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:30:00.427670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.427650 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:00.428964 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.428941 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4927ea-a695-4eaa-8443-2e83a61db882-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.429489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.429473 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc662d85-9037-48e9-b509-0c2ace6fb267-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.473646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.473605 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpbr\" (UniqueName: \"kubernetes.io/projected/9c4927ea-a695-4eaa-8443-2e83a61db882-kube-api-access-hqpbr\") pod \"maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.572374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.570886 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-openshift-ca\") pod \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " Oct 30 18:30:00.572374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.570956 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-xks-ca\") pod \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " Oct 30 18:30:00.572374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.571065 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-kube-api-access-8ncz9\" (UniqueName: \"kubernetes.io/projected/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-kube-api-access-8ncz9\") pod \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\" (UID: \"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795\") " Oct 30 18:30:00.572374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.572197 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-openshift-ca" pod "25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" (UID: "25b1fbfd-2dfc-4617-81dc-9eb1ded2b795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:00.621624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.619812 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-xks-ca" pod "25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" (UID: "25b1fbfd-2dfc-4617-81dc-9eb1ded2b795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:00.621624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.619997 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:30:00.621624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.620048 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 18:30:00.621624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.621127 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqrh\" (UniqueName: \"kubernetes.io/projected/cc662d85-9037-48e9-b509-0c2ace6fb267-kube-api-access-mmqrh\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.622760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.622602 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:00.623188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.623165 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-kube-api-access-8ncz9" pod "25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" (UID: "25b1fbfd-2dfc-4617-81dc-9eb1ded2b795"). InnerVolumeSpecName "kube-api-access-8ncz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:00.624200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.624175 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzt4\" (UniqueName: \"kubernetes.io/projected/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-kube-api-access-nzzt4\") pod \"maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:00.629946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.627211 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75"] Oct 30 18:30:00.630442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.630225 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:00.694308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.684563 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:00.694308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.684653 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:00.694308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.684670 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ncz9\" (UniqueName: \"kubernetes.io/projected/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795-kube-api-access-8ncz9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:00.731581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.731327 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:00.883837 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:00.872263 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:01.086843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:01.083207 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" event={"ID":"25b1fbfd-2dfc-4617-81dc-9eb1ded2b795","Type":"ContainerDied","Data":"2be8267a984301d11588c9ef753e4d774213f345299fbcb1c50438e4d2af2f1e"} Oct 30 18:30:01.086843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:01.083272 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be8267a984301d11588c9ef753e4d774213f345299fbcb1c50438e4d2af2f1e" Oct 30 18:30:01.086843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:01.083346 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9" Oct 30 18:30:01.124257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:01.124225 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f"] Oct 30 18:30:01.568297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:01.568255 4582 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 18:30:01.568607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:01.568548 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 18:30:02.543585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.543548 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" event={"ID":"f192d012-e8bc-496b-a03c-5cfb8820f38d","Type":"ContainerStarted","Data":"de7535aea84192a80e4d276c8c68de9c4769a61082a2dc21f29951c0f008c1e9"} Oct 30 18:30:02.544262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.544216 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" event={"ID":"f192d012-e8bc-496b-a03c-5cfb8820f38d","Type":"ContainerStarted","Data":"a31a954ea9c7861d02146e1e42e21a0040a52fa014f8bbe65c6250bf808cf23e"} Oct 30 18:30:02.547754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.547730 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" event={"ID":"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2","Type":"ContainerStarted","Data":"ac90b0ee654480bbb7591717b563d951f547527095e41bdee306fccf4c5f348c"} Oct 30 18:30:02.548463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.548423 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" event={"ID":"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2","Type":"ContainerStarted","Data":"feeb64aa5c52794bb4d5feada54649174401df8f365226d3e0ffeea4d2f172a0"} Oct 30 18:30:02.647700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.647600 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w"] Oct 30 18:30:02.662917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.661856 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" podStartSLOduration=3.66183899 podStartE2EDuration="3.66183899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:02.634029253 +0000 UTC m=+1899.458423570" watchObservedRunningTime="2026-10-30 18:30:02.66183899 +0000 UTC m=+1899.486233281" Oct 30 18:30:02.679481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.679423 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" podStartSLOduration=3.679405693 podStartE2EDuration="3.679405693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:02.668901194 +0000 UTC m=+1899.493295502" watchObservedRunningTime="2026-10-30 18:30:02.679405693 +0000 UTC m=+1899.503799989" Oct 30 18:30:02.927856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.927706 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:30:02.929055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.928907 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:30:02.931891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.931856 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:30:02.941947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:02.941879 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7"] Oct 30 18:30:03.014520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.014491 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw"] Oct 30 18:30:03.017605 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:03.017578 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5205b0ec_26f2_49fa_a270_1eca15bbc7b9.slice/crio-be6bd2eaf120f53b0282fda66edebd9f3ff8a407a0738b6ae1ad5b3a2d82d39a WatchSource:0}: Error finding container be6bd2eaf120f53b0282fda66edebd9f3ff8a407a0738b6ae1ad5b3a2d82d39a: Status 404 returned error can't find the container with id be6bd2eaf120f53b0282fda66edebd9f3ff8a407a0738b6ae1ad5b3a2d82d39a Oct 30 18:30:03.505175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.505089 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99"] Oct 30 18:30:03.505676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.505659 4582 cpu_manager.go:414] "containerMap: removing container" podUID="25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" containerName="revoke-keys" Oct 30 18:30:03.505676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.505673 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" containerName="revoke-keys" Oct 30 18:30:03.505881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.505788 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" containerName="revoke-keys" Oct 30 18:30:03.509258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.509237 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.530175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.530133 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99"] Oct 30 18:30:03.557998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.557963 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" event={"ID":"cc662d85-9037-48e9-b509-0c2ace6fb267","Type":"ContainerStarted","Data":"b29911f45ac70b7285dc62e483a13327b4df8b6769f7f37b9d5f13d7450c4409"} Oct 30 18:30:03.558503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.558025 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" event={"ID":"cc662d85-9037-48e9-b509-0c2ace6fb267","Type":"ContainerStarted","Data":"044c0aeffe0160434f3fe0a190f60621caa4b8e1ec2385d91c5c88e827adf7bc"} Oct 30 18:30:03.560385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.560353 4582 generic.go:356] "Generic (PLEG): container finished" podID="9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" containerID="ac90b0ee654480bbb7591717b563d951f547527095e41bdee306fccf4c5f348c" exitCode=0 Oct 30 18:30:03.560501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.560437 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" event={"ID":"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2","Type":"ContainerDied","Data":"ac90b0ee654480bbb7591717b563d951f547527095e41bdee306fccf4c5f348c"} Oct 30 18:30:03.562598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.562572 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerStarted","Data":"8bb0050275ec5a4b23f69b9438445795a009cce6aca81f801aed9deaaf294805"} Oct 30 18:30:03.562709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.562620 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerStarted","Data":"be6bd2eaf120f53b0282fda66edebd9f3ff8a407a0738b6ae1ad5b3a2d82d39a"} Oct 30 18:30:03.564669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.564578 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" event={"ID":"9c4927ea-a695-4eaa-8443-2e83a61db882","Type":"ContainerStarted","Data":"6bfd17844c2636763b45eefb1f3b4ccceb18dd0bc2ef3a6dfe5a2b01c3d273a0"} Oct 30 18:30:03.564669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.564637 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" event={"ID":"9c4927ea-a695-4eaa-8443-2e83a61db882","Type":"ContainerStarted","Data":"d663ffb96bb7024eeb545eac967953b35bbb7b421df0d522bb81a85e20f55c3b"} Oct 30 18:30:03.566678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.566656 4582 generic.go:356] "Generic (PLEG): container finished" podID="f192d012-e8bc-496b-a03c-5cfb8820f38d" containerID="de7535aea84192a80e4d276c8c68de9c4769a61082a2dc21f29951c0f008c1e9" exitCode=0 Oct 30 18:30:03.566805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.566741 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" event={"ID":"f192d012-e8bc-496b-a03c-5cfb8820f38d","Type":"ContainerDied","Data":"de7535aea84192a80e4d276c8c68de9c4769a61082a2dc21f29951c0f008c1e9"} Oct 30 18:30:03.592625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.592583 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" podStartSLOduration=3.592568284 podStartE2EDuration="3.592568284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:03.589771821 +0000 UTC m=+1900.414166144" watchObservedRunningTime="2026-10-30 18:30:03.592568284 +0000 UTC m=+1900.416962577" Oct 30 18:30:03.636622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.636585 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.636622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.636624 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8lt\" (UniqueName: \"kubernetes.io/projected/1dfb9ac7-cf08-461e-a033-16d81ea29094-kube-api-access-ll8lt\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.636860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.636651 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.653700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.653633 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" podStartSLOduration=3.653616557 podStartE2EDuration="3.653616557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:03.652506595 +0000 UTC m=+1900.476900890" watchObservedRunningTime="2026-10-30 18:30:03.653616557 +0000 UTC m=+1900.478010853" Oct 30 18:30:03.719338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.719287 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" podStartSLOduration=3.719274115 podStartE2EDuration="3.719274115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:03.715523661 +0000 UTC m=+1900.539917952" watchObservedRunningTime="2026-10-30 18:30:03.719274115 +0000 UTC m=+1900.543668438" Oct 30 18:30:03.738349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.738318 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.738349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.738351 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8lt\" (UniqueName: \"kubernetes.io/projected/1dfb9ac7-cf08-461e-a033-16d81ea29094-kube-api-access-ll8lt\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.738607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.738378 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.739102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.739073 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.740901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.740881 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.754762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.754733 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8lt\" (UniqueName: \"kubernetes.io/projected/1dfb9ac7-cf08-461e-a033-16d81ea29094-kube-api-access-ll8lt\") pod \"maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.823260 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.823184 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:03.977659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:03.977627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:04.153855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.153806 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99"] Oct 30 18:30:04.575150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.575111 4582 generic.go:356] "Generic (PLEG): container finished" podID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerID="b29911f45ac70b7285dc62e483a13327b4df8b6769f7f37b9d5f13d7450c4409" exitCode=60 Oct 30 18:30:04.575613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.575197 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" event={"ID":"cc662d85-9037-48e9-b509-0c2ace6fb267","Type":"ContainerDied","Data":"b29911f45ac70b7285dc62e483a13327b4df8b6769f7f37b9d5f13d7450c4409"} Oct 30 18:30:04.575613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.575452 4582 scope.go:122] "RemoveContainer" containerID="b29911f45ac70b7285dc62e483a13327b4df8b6769f7f37b9d5f13d7450c4409" Oct 30 18:30:04.577164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.577104 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" event={"ID":"1dfb9ac7-cf08-461e-a033-16d81ea29094","Type":"ContainerStarted","Data":"0456fbec8a43245448332f100b33169241e9aafebdc3f168c9084e715918ee2f"} Oct 30 18:30:04.577164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.577141 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" event={"ID":"1dfb9ac7-cf08-461e-a033-16d81ea29094","Type":"ContainerStarted","Data":"147c4d42687baf01f3dcaf742752189649bc2f8ef5c2d543d610012aaabb3df1"} Oct 30 18:30:04.578897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.578877 4582 generic.go:356] "Generic (PLEG): container finished" podID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerID="8bb0050275ec5a4b23f69b9438445795a009cce6aca81f801aed9deaaf294805" exitCode=60 Oct 30 18:30:04.578989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.578966 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerDied","Data":"8bb0050275ec5a4b23f69b9438445795a009cce6aca81f801aed9deaaf294805"} Oct 30 18:30:04.579173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.579153 4582 scope.go:122] "RemoveContainer" containerID="8bb0050275ec5a4b23f69b9438445795a009cce6aca81f801aed9deaaf294805" Oct 30 18:30:04.580763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.580741 4582 generic.go:356] "Generic (PLEG): container finished" podID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerID="6bfd17844c2636763b45eefb1f3b4ccceb18dd0bc2ef3a6dfe5a2b01c3d273a0" exitCode=60 Oct 30 18:30:04.580911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.580838 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" event={"ID":"9c4927ea-a695-4eaa-8443-2e83a61db882","Type":"ContainerDied","Data":"6bfd17844c2636763b45eefb1f3b4ccceb18dd0bc2ef3a6dfe5a2b01c3d273a0"} Oct 30 18:30:04.581057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.581040 4582 scope.go:122] "RemoveContainer" containerID="6bfd17844c2636763b45eefb1f3b4ccceb18dd0bc2ef3a6dfe5a2b01c3d273a0" Oct 30 18:30:04.716120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.716010 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" podStartSLOduration=1.715993922 podStartE2EDuration="1.715993922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:04.711468181 +0000 UTC m=+1901.535862513" watchObservedRunningTime="2026-10-30 18:30:04.715993922 +0000 UTC m=+1901.540388215" Oct 30 18:30:04.768642 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.768616 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:04.771026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.771007 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:30:04.953970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.953939 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-openshift-ca\") pod \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " Oct 30 18:30:04.954138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.954057 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f192d012-e8bc-496b-a03c-5cfb8820f38d-kube-api-access-p6mfp\" (UniqueName: \"kubernetes.io/projected/f192d012-e8bc-496b-a03c-5cfb8820f38d-kube-api-access-p6mfp\") pod \"f192d012-e8bc-496b-a03c-5cfb8820f38d\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " Oct 30 18:30:04.954138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.954113 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-kube-api-access-92xvp\" (UniqueName: \"kubernetes.io/projected/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-kube-api-access-92xvp\") pod \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " Oct 30 18:30:04.954269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.954182 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-xks-ca\") pod \"f192d012-e8bc-496b-a03c-5cfb8820f38d\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " Oct 30 18:30:04.954269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.954261 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-openshift-ca\") pod \"f192d012-e8bc-496b-a03c-5cfb8820f38d\" (UID: \"f192d012-e8bc-496b-a03c-5cfb8820f38d\") " Oct 30 18:30:04.954391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.954300 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-openshift-ca" pod "9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" (UID: "9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:04.954391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.954318 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-xks-ca\") pod \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\" (UID: \"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2\") " Oct 30 18:30:04.954750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.954644 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-openshift-ca" pod "f192d012-e8bc-496b-a03c-5cfb8820f38d" (UID: "f192d012-e8bc-496b-a03c-5cfb8820f38d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:04.955157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.955016 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.955157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.955039 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.956866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.956804 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f192d012-e8bc-496b-a03c-5cfb8820f38d-kube-api-access-p6mfp" pod "f192d012-e8bc-496b-a03c-5cfb8820f38d" (UID: "f192d012-e8bc-496b-a03c-5cfb8820f38d"). InnerVolumeSpecName "kube-api-access-p6mfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:04.956994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.956930 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-kube-api-access-92xvp" pod "9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" (UID: "9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2"). InnerVolumeSpecName "kube-api-access-92xvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:04.957112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.957066 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-xks-ca" pod "9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" (UID: "9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:04.957399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:04.957370 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-xks-ca" pod "f192d012-e8bc-496b-a03c-5cfb8820f38d" (UID: "f192d012-e8bc-496b-a03c-5cfb8820f38d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:05.056767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.056744 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6mfp\" (UniqueName: \"kubernetes.io/projected/f192d012-e8bc-496b-a03c-5cfb8820f38d-kube-api-access-p6mfp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:05.056767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.056769 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92xvp\" (UniqueName: \"kubernetes.io/projected/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-kube-api-access-92xvp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:05.056983 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.056779 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f192d012-e8bc-496b-a03c-5cfb8820f38d-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:05.056983 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.056788 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:05.591259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.591223 4582 generic.go:356] "Generic (PLEG): container finished" podID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerID="f4b6c51680cbcac5e970a7ed35ee0434bff4cb668a18f7915f8f494a17eb9a3b" exitCode=60 Oct 30 18:30:05.591712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.591302 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" event={"ID":"9c4927ea-a695-4eaa-8443-2e83a61db882","Type":"ContainerDied","Data":"f4b6c51680cbcac5e970a7ed35ee0434bff4cb668a18f7915f8f494a17eb9a3b"} Oct 30 18:30:05.591712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.591386 4582 scope.go:122] "RemoveContainer" containerID="6bfd17844c2636763b45eefb1f3b4ccceb18dd0bc2ef3a6dfe5a2b01c3d273a0" Oct 30 18:30:05.591712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.591523 4582 scope.go:122] "RemoveContainer" containerID="f4b6c51680cbcac5e970a7ed35ee0434bff4cb668a18f7915f8f494a17eb9a3b" Oct 30 18:30:05.591867 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:05.591784 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7_odh-ai-gateway-infra(9c4927ea-a695-4eaa-8443-2e83a61db882)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" Oct 30 18:30:05.592966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.592942 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" event={"ID":"f192d012-e8bc-496b-a03c-5cfb8820f38d","Type":"ContainerDied","Data":"a31a954ea9c7861d02146e1e42e21a0040a52fa014f8bbe65c6250bf808cf23e"} Oct 30 18:30:05.592966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.592963 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f" Oct 30 18:30:05.593122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.592990 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31a954ea9c7861d02146e1e42e21a0040a52fa014f8bbe65c6250bf808cf23e" Oct 30 18:30:05.594659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.594640 4582 generic.go:356] "Generic (PLEG): container finished" podID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerID="525565efa67be8a05c96a158a6a143a1dc560ec61c49c880316695d12420327f" exitCode=60 Oct 30 18:30:05.594747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.594718 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" event={"ID":"cc662d85-9037-48e9-b509-0c2ace6fb267","Type":"ContainerDied","Data":"525565efa67be8a05c96a158a6a143a1dc560ec61c49c880316695d12420327f"} Oct 30 18:30:05.594938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.594924 4582 scope.go:122] "RemoveContainer" containerID="525565efa67be8a05c96a158a6a143a1dc560ec61c49c880316695d12420327f" Oct 30 18:30:05.595191 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:05.595172 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w_odh-ai-gateway-infra(cc662d85-9037-48e9-b509-0c2ace6fb267)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" Oct 30 18:30:05.596287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.596264 4582 generic.go:356] "Generic (PLEG): container finished" podID="1dfb9ac7-cf08-461e-a033-16d81ea29094" containerID="0456fbec8a43245448332f100b33169241e9aafebdc3f168c9084e715918ee2f" exitCode=0 Oct 30 18:30:05.596375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.596359 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" event={"ID":"1dfb9ac7-cf08-461e-a033-16d81ea29094","Type":"ContainerDied","Data":"0456fbec8a43245448332f100b33169241e9aafebdc3f168c9084e715918ee2f"} Oct 30 18:30:05.597573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.597544 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" event={"ID":"9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2","Type":"ContainerDied","Data":"feeb64aa5c52794bb4d5feada54649174401df8f365226d3e0ffeea4d2f172a0"} Oct 30 18:30:05.597662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.597582 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feeb64aa5c52794bb4d5feada54649174401df8f365226d3e0ffeea4d2f172a0" Oct 30 18:30:05.597662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.597552 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75" Oct 30 18:30:05.599229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.599208 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerStarted","Data":"e6b15fd129f5a843aa8131d8d7d0776f36cad7a402c21e2f9a9fd5ee9118bb5e"} Oct 30 18:30:05.614739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.614720 4582 scope.go:122] "RemoveContainer" containerID="b29911f45ac70b7285dc62e483a13327b4df8b6769f7f37b9d5f13d7450c4409" Oct 30 18:30:05.970737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:05.970707 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:06.225589 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:06.225501 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:30:06.610514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.610438 4582 scope.go:122] "RemoveContainer" containerID="f4b6c51680cbcac5e970a7ed35ee0434bff4cb668a18f7915f8f494a17eb9a3b" Oct 30 18:30:06.610962 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:06.610789 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7_odh-ai-gateway-infra(9c4927ea-a695-4eaa-8443-2e83a61db882)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" Oct 30 18:30:06.612389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.612364 4582 scope.go:122] "RemoveContainer" containerID="525565efa67be8a05c96a158a6a143a1dc560ec61c49c880316695d12420327f" Oct 30 18:30:06.612702 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:06.612678 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w_odh-ai-gateway-infra(cc662d85-9037-48e9-b509-0c2ace6fb267)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" Oct 30 18:30:06.613960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.613935 4582 generic.go:356] "Generic (PLEG): container finished" podID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerID="e6b15fd129f5a843aa8131d8d7d0776f36cad7a402c21e2f9a9fd5ee9118bb5e" exitCode=60 Oct 30 18:30:06.614093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.614017 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerDied","Data":"e6b15fd129f5a843aa8131d8d7d0776f36cad7a402c21e2f9a9fd5ee9118bb5e"} Oct 30 18:30:06.614162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.614116 4582 scope.go:122] "RemoveContainer" containerID="8bb0050275ec5a4b23f69b9438445795a009cce6aca81f801aed9deaaf294805" Oct 30 18:30:06.614237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.614224 4582 scope.go:122] "RemoveContainer" containerID="e6b15fd129f5a843aa8131d8d7d0776f36cad7a402c21e2f9a9fd5ee9118bb5e" Oct 30 18:30:06.614692 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:06.614474 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw_odh-ai-gateway-infra(5205b0ec-26f2-49fa-a270-1eca15bbc7b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" Oct 30 18:30:06.788066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.788035 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:06.976893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.976858 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-openshift-ca\") pod \"1dfb9ac7-cf08-461e-a033-16d81ea29094\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " Oct 30 18:30:06.977083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.976957 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfb9ac7-cf08-461e-a033-16d81ea29094-kube-api-access-ll8lt\" (UniqueName: \"kubernetes.io/projected/1dfb9ac7-cf08-461e-a033-16d81ea29094-kube-api-access-ll8lt\") pod \"1dfb9ac7-cf08-461e-a033-16d81ea29094\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " Oct 30 18:30:06.977083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.976982 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-xks-ca\") pod \"1dfb9ac7-cf08-461e-a033-16d81ea29094\" (UID: \"1dfb9ac7-cf08-461e-a033-16d81ea29094\") " Oct 30 18:30:06.977277 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.977249 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-openshift-ca" pod "1dfb9ac7-cf08-461e-a033-16d81ea29094" (UID: "1dfb9ac7-cf08-461e-a033-16d81ea29094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:06.977459 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.977442 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:06.979013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.978982 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-xks-ca" pod "1dfb9ac7-cf08-461e-a033-16d81ea29094" (UID: "1dfb9ac7-cf08-461e-a033-16d81ea29094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:06.979120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:06.979095 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfb9ac7-cf08-461e-a033-16d81ea29094-kube-api-access-ll8lt" pod "1dfb9ac7-cf08-461e-a033-16d81ea29094" (UID: "1dfb9ac7-cf08-461e-a033-16d81ea29094"). InnerVolumeSpecName "kube-api-access-ll8lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:07.078744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:07.078709 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll8lt\" (UniqueName: \"kubernetes.io/projected/1dfb9ac7-cf08-461e-a033-16d81ea29094-kube-api-access-ll8lt\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:07.078744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:07.078746 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dfb9ac7-cf08-461e-a033-16d81ea29094-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:07.624248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:07.624216 4582 scope.go:122] "RemoveContainer" containerID="e6b15fd129f5a843aa8131d8d7d0776f36cad7a402c21e2f9a9fd5ee9118bb5e" Oct 30 18:30:07.624693 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:07.624510 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw_odh-ai-gateway-infra(5205b0ec-26f2-49fa-a270-1eca15bbc7b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" Oct 30 18:30:07.625505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:07.625483 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" event={"ID":"1dfb9ac7-cf08-461e-a033-16d81ea29094","Type":"ContainerDied","Data":"147c4d42687baf01f3dcaf742752189649bc2f8ef5c2d543d610012aaabb3df1"} Oct 30 18:30:07.625505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:07.625495 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99" Oct 30 18:30:07.625647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:07.625526 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147c4d42687baf01f3dcaf742752189649bc2f8ef5c2d543d610012aaabb3df1" Oct 30 18:30:07.861886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:07.861852 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:09.567874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.567804 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:09.903860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.903812 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d"] Oct 30 18:30:09.904407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904386 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" containerName="revoke-keys" Oct 30 18:30:09.904407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904403 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" containerName="revoke-keys" Oct 30 18:30:09.904607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904423 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f192d012-e8bc-496b-a03c-5cfb8820f38d" containerName="revoke-keys" Oct 30 18:30:09.904607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904429 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f192d012-e8bc-496b-a03c-5cfb8820f38d" containerName="revoke-keys" Oct 30 18:30:09.904607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904456 4582 cpu_manager.go:414] "containerMap: removing container" podUID="1dfb9ac7-cf08-461e-a033-16d81ea29094" containerName="revoke-keys" Oct 30 18:30:09.904607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904461 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfb9ac7-cf08-461e-a033-16d81ea29094" containerName="revoke-keys" Oct 30 18:30:09.904607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904574 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" containerName="revoke-keys" Oct 30 18:30:09.904607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904588 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f192d012-e8bc-496b-a03c-5cfb8820f38d" containerName="revoke-keys" Oct 30 18:30:09.904607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.904598 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfb9ac7-cf08-461e-a033-16d81ea29094" containerName="revoke-keys" Oct 30 18:30:09.908380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.908362 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:09.921644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.921621 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:30:09.936553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:09.936532 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d"] Oct 30 18:30:10.015909 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.015869 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.016065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.015962 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.016191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.016175 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstzz\" (UniqueName: \"kubernetes.io/projected/64af1e67-48a2-4102-ad6b-3775ceb9fe48-kube-api-access-sstzz\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.117140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.117106 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstzz\" (UniqueName: \"kubernetes.io/projected/64af1e67-48a2-4102-ad6b-3775ceb9fe48-kube-api-access-sstzz\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.117329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.117179 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.117329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.117206 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.117999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.117968 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.119631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.119598 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.133580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.133553 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstzz\" (UniqueName: \"kubernetes.io/projected/64af1e67-48a2-4102-ad6b-3775ceb9fe48-kube-api-access-sstzz\") pod \"maas-api-revoke-sub-8767f707-rkq5d\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.221399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.221294 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:10.518978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.518952 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d"] Oct 30 18:30:10.522024 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:10.521988 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64af1e67_48a2_4102_ad6b_3775ceb9fe48.slice/crio-08833f577ee8d6a3ca355b206b5bad97e7ea2b20617b2f0259d7012c32011dca WatchSource:0}: Error finding container 08833f577ee8d6a3ca355b206b5bad97e7ea2b20617b2f0259d7012c32011dca: Status 404 returned error can't find the container with id 08833f577ee8d6a3ca355b206b5bad97e7ea2b20617b2f0259d7012c32011dca Oct 30 18:30:10.653435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:10.653398 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" event={"ID":"64af1e67-48a2-4102-ad6b-3775ceb9fe48","Type":"ContainerStarted","Data":"08833f577ee8d6a3ca355b206b5bad97e7ea2b20617b2f0259d7012c32011dca"} Oct 30 18:30:11.260472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:11.260443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:11.661803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:11.661767 4582 generic.go:356] "Generic (PLEG): container finished" podID="64af1e67-48a2-4102-ad6b-3775ceb9fe48" containerID="a18ccc7785f9762aedf2b6d9784fe828b557bcfd12f6b79b08a3e208a4334623" exitCode=0 Oct 30 18:30:11.662226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:11.661852 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" event={"ID":"64af1e67-48a2-4102-ad6b-3775ceb9fe48","Type":"ContainerDied","Data":"a18ccc7785f9762aedf2b6d9784fe828b557bcfd12f6b79b08a3e208a4334623"} Oct 30 18:30:12.868332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:12.868299 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:12.983903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:12.983094 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-openshift-ca\") pod \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " Oct 30 18:30:12.983903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:12.983266 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-xks-ca\") pod \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " Oct 30 18:30:12.983903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:12.983328 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64af1e67-48a2-4102-ad6b-3775ceb9fe48-kube-api-access-sstzz\" (UniqueName: \"kubernetes.io/projected/64af1e67-48a2-4102-ad6b-3775ceb9fe48-kube-api-access-sstzz\") pod \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\" (UID: \"64af1e67-48a2-4102-ad6b-3775ceb9fe48\") " Oct 30 18:30:13.009077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:12.984581 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-openshift-ca" pod "64af1e67-48a2-4102-ad6b-3775ceb9fe48" (UID: "64af1e67-48a2-4102-ad6b-3775ceb9fe48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:13.009937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.009912 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64af1e67-48a2-4102-ad6b-3775ceb9fe48-kube-api-access-sstzz" pod "64af1e67-48a2-4102-ad6b-3775ceb9fe48" (UID: "64af1e67-48a2-4102-ad6b-3775ceb9fe48"). InnerVolumeSpecName "kube-api-access-sstzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:13.013226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.013198 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-xks-ca" pod "64af1e67-48a2-4102-ad6b-3775ceb9fe48" (UID: "64af1e67-48a2-4102-ad6b-3775ceb9fe48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:13.085675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.085622 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:13.085675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.085666 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64af1e67-48a2-4102-ad6b-3775ceb9fe48-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:13.085675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.085682 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sstzz\" (UniqueName: \"kubernetes.io/projected/64af1e67-48a2-4102-ad6b-3775ceb9fe48-kube-api-access-sstzz\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:13.486844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.469747 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:13.720862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.718132 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" event={"ID":"64af1e67-48a2-4102-ad6b-3775ceb9fe48","Type":"ContainerDied","Data":"08833f577ee8d6a3ca355b206b5bad97e7ea2b20617b2f0259d7012c32011dca"} Oct 30 18:30:13.720862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.718193 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08833f577ee8d6a3ca355b206b5bad97e7ea2b20617b2f0259d7012c32011dca" Oct 30 18:30:13.720862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:13.718258 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d" Oct 30 18:30:14.064562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.064251 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:14.157476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.157417 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:14.959281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.959245 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg"] Oct 30 18:30:14.959804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.959790 4582 cpu_manager.go:414] "containerMap: removing container" podUID="64af1e67-48a2-4102-ad6b-3775ceb9fe48" containerName="revoke-keys" Oct 30 18:30:14.959900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.959804 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64af1e67-48a2-4102-ad6b-3775ceb9fe48" containerName="revoke-keys" Oct 30 18:30:14.959961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.959928 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="64af1e67-48a2-4102-ad6b-3775ceb9fe48" containerName="revoke-keys" Oct 30 18:30:14.963695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.963674 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:14.982353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.982331 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg"] Oct 30 18:30:14.994073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:14.994049 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:30:15.005232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.005197 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:15.027233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.027201 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26jh\" (UniqueName: \"kubernetes.io/projected/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-kube-api-access-h26jh\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.027349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.027257 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.027482 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.027462 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.128397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.128366 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.128776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.128505 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.128776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.128557 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26jh\" (UniqueName: \"kubernetes.io/projected/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-kube-api-access-h26jh\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.129047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.129019 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.130731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.130706 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.145629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.145604 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26jh\" (UniqueName: \"kubernetes.io/projected/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-kube-api-access-h26jh\") pod \"maas-api-revoke-sub-a3110575-86xrg\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.276453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.276381 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:15.586266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.586237 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg"] Oct 30 18:30:15.588520 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:15.588491 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d1f3cd_0cd7_4583_9273_dfd4fdbdfac1.slice/crio-48679e55296750af211bfc50120ab10cf40e1e2373749cdfece2ce14e8419fe9 WatchSource:0}: Error finding container 48679e55296750af211bfc50120ab10cf40e1e2373749cdfece2ce14e8419fe9: Status 404 returned error can't find the container with id 48679e55296750af211bfc50120ab10cf40e1e2373749cdfece2ce14e8419fe9 Oct 30 18:30:15.624925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.624892 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:15.736753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:15.736718 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" event={"ID":"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1","Type":"ContainerStarted","Data":"48679e55296750af211bfc50120ab10cf40e1e2373749cdfece2ce14e8419fe9"} Oct 30 18:30:16.746236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:16.746198 4582 generic.go:356] "Generic (PLEG): container finished" podID="b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" containerID="c5a441a49b6e7da48965a9ef45ae20546ee24af3af994454ee36d782529d8f32" exitCode=0 Oct 30 18:30:16.746605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:16.746284 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" event={"ID":"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1","Type":"ContainerDied","Data":"c5a441a49b6e7da48965a9ef45ae20546ee24af3af994454ee36d782529d8f32"} Oct 30 18:30:17.293541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.293506 4582 scope.go:122] "RemoveContainer" containerID="525565efa67be8a05c96a158a6a143a1dc560ec61c49c880316695d12420327f" Oct 30 18:30:17.358639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.358613 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:17.755662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.755631 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" event={"ID":"cc662d85-9037-48e9-b509-0c2ace6fb267","Type":"ContainerStarted","Data":"cc14161c6e7466144ef8f5f7c529ccf0e9b0dad9f3998f9934e55de80f13f49b"} Oct 30 18:30:17.907412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.907351 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:17.959991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.959964 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-kube-api-access-h26jh\" (UniqueName: \"kubernetes.io/projected/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-kube-api-access-h26jh\") pod \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " Oct 30 18:30:17.960149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.960017 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-openshift-ca\") pod \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " Oct 30 18:30:17.960149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.960070 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-xks-ca\") pod \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\" (UID: \"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1\") " Oct 30 18:30:17.960445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.960419 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-openshift-ca" pod "b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" (UID: "b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:17.960655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.960639 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.962167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.962145 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-xks-ca" pod "b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" (UID: "b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:17.962274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:17.962199 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-kube-api-access-h26jh" pod "b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" (UID: "b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1"). InnerVolumeSpecName "kube-api-access-h26jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:18.062147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.062113 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h26jh\" (UniqueName: \"kubernetes.io/projected/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-kube-api-access-h26jh\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.062147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.062141 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.765918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.765883 4582 generic.go:356] "Generic (PLEG): container finished" podID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerID="cc14161c6e7466144ef8f5f7c529ccf0e9b0dad9f3998f9934e55de80f13f49b" exitCode=60 Oct 30 18:30:18.766326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.765957 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" event={"ID":"cc662d85-9037-48e9-b509-0c2ace6fb267","Type":"ContainerDied","Data":"cc14161c6e7466144ef8f5f7c529ccf0e9b0dad9f3998f9934e55de80f13f49b"} Oct 30 18:30:18.766326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.766014 4582 scope.go:122] "RemoveContainer" containerID="525565efa67be8a05c96a158a6a143a1dc560ec61c49c880316695d12420327f" Oct 30 18:30:18.766581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.766561 4582 scope.go:122] "RemoveContainer" containerID="cc14161c6e7466144ef8f5f7c529ccf0e9b0dad9f3998f9934e55de80f13f49b" Oct 30 18:30:18.766938 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:18.766918 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w_odh-ai-gateway-infra(cc662d85-9037-48e9-b509-0c2ace6fb267)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" Oct 30 18:30:18.767625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.767607 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" event={"ID":"b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1","Type":"ContainerDied","Data":"48679e55296750af211bfc50120ab10cf40e1e2373749cdfece2ce14e8419fe9"} Oct 30 18:30:18.767690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.767643 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48679e55296750af211bfc50120ab10cf40e1e2373749cdfece2ce14e8419fe9" Oct 30 18:30:18.767690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.767653 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg" Oct 30 18:30:18.802487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:18.802461 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w"] Oct 30 18:30:19.167088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.167061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:19.292971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.292943 4582 scope.go:122] "RemoveContainer" containerID="f4b6c51680cbcac5e970a7ed35ee0434bff4cb668a18f7915f8f494a17eb9a3b" Oct 30 18:30:19.777697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.777599 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" event={"ID":"9c4927ea-a695-4eaa-8443-2e83a61db882","Type":"ContainerStarted","Data":"d0b7fce62d79dcd6e94c395cca08d2e56372238349979ba6d3aed9415a04bc51"} Oct 30 18:30:19.937094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.937065 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:19.983418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.983383 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc662d85-9037-48e9-b509-0c2ace6fb267-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc662d85-9037-48e9-b509-0c2ace6fb267-maas-api-service-ca\") pod \"cc662d85-9037-48e9-b509-0c2ace6fb267\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " Oct 30 18:30:19.983577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.983523 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc662d85-9037-48e9-b509-0c2ace6fb267-kube-api-access-mmqrh\" (UniqueName: \"kubernetes.io/projected/cc662d85-9037-48e9-b509-0c2ace6fb267-kube-api-access-mmqrh\") pod \"cc662d85-9037-48e9-b509-0c2ace6fb267\" (UID: \"cc662d85-9037-48e9-b509-0c2ace6fb267\") " Oct 30 18:30:19.983781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.983755 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc662d85-9037-48e9-b509-0c2ace6fb267-maas-api-service-ca" pod "cc662d85-9037-48e9-b509-0c2ace6fb267" (UID: "cc662d85-9037-48e9-b509-0c2ace6fb267"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:19.984148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.984125 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc662d85-9037-48e9-b509-0c2ace6fb267-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:19.985647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:19.985622 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc662d85-9037-48e9-b509-0c2ace6fb267-kube-api-access-mmqrh" pod "cc662d85-9037-48e9-b509-0c2ace6fb267" (UID: "cc662d85-9037-48e9-b509-0c2ace6fb267"). InnerVolumeSpecName "kube-api-access-mmqrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:20.085096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.085073 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmqrh\" (UniqueName: \"kubernetes.io/projected/cc662d85-9037-48e9-b509-0c2ace6fb267-kube-api-access-mmqrh\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:20.293005 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.292978 4582 scope.go:122] "RemoveContainer" containerID="e6b15fd129f5a843aa8131d8d7d0776f36cad7a402c21e2f9a9fd5ee9118bb5e" Oct 30 18:30:20.787634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.787609 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" Oct 30 18:30:20.788160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.787643 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w" event={"ID":"cc662d85-9037-48e9-b509-0c2ace6fb267","Type":"ContainerDied","Data":"044c0aeffe0160434f3fe0a190f60621caa4b8e1ec2385d91c5c88e827adf7bc"} Oct 30 18:30:20.788160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.787703 4582 scope.go:122] "RemoveContainer" containerID="cc14161c6e7466144ef8f5f7c529ccf0e9b0dad9f3998f9934e55de80f13f49b" Oct 30 18:30:20.790217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.790191 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerStarted","Data":"2fea66928f043455050a51a754d89b553485d3c0f38280054cc7448471792ff3"} Oct 30 18:30:20.792542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.792511 4582 generic.go:356] "Generic (PLEG): container finished" podID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerID="d0b7fce62d79dcd6e94c395cca08d2e56372238349979ba6d3aed9415a04bc51" exitCode=60 Oct 30 18:30:20.792679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.792585 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" event={"ID":"9c4927ea-a695-4eaa-8443-2e83a61db882","Type":"ContainerDied","Data":"d0b7fce62d79dcd6e94c395cca08d2e56372238349979ba6d3aed9415a04bc51"} Oct 30 18:30:20.793014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.792986 4582 scope.go:122] "RemoveContainer" containerID="d0b7fce62d79dcd6e94c395cca08d2e56372238349979ba6d3aed9415a04bc51" Oct 30 18:30:20.793278 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:20.793255 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7_odh-ai-gateway-infra(9c4927ea-a695-4eaa-8443-2e83a61db882)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" Oct 30 18:30:20.801162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.801092 4582 scope.go:122] "RemoveContainer" containerID="f4b6c51680cbcac5e970a7ed35ee0434bff4cb668a18f7915f8f494a17eb9a3b" Oct 30 18:30:20.822770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.822745 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7"] Oct 30 18:30:20.897275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.897243 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w"] Oct 30 18:30:20.902119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:20.902094 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889750-zf97w"] Oct 30 18:30:21.047694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.047584 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:21.301212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.301125 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" path="/var/lib/kubelet/pods/cc662d85-9037-48e9-b509-0c2ace6fb267/volumes" Oct 30 18:30:21.802361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.802328 4582 generic.go:356] "Generic (PLEG): container finished" podID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerID="2fea66928f043455050a51a754d89b553485d3c0f38280054cc7448471792ff3" exitCode=60 Oct 30 18:30:21.802784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.802397 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerDied","Data":"2fea66928f043455050a51a754d89b553485d3c0f38280054cc7448471792ff3"} Oct 30 18:30:21.802784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.802458 4582 scope.go:122] "RemoveContainer" containerID="e6b15fd129f5a843aa8131d8d7d0776f36cad7a402c21e2f9a9fd5ee9118bb5e" Oct 30 18:30:21.802894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.802875 4582 scope.go:122] "RemoveContainer" containerID="2fea66928f043455050a51a754d89b553485d3c0f38280054cc7448471792ff3" Oct 30 18:30:21.803266 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:21.803236 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw_odh-ai-gateway-infra(5205b0ec-26f2-49fa-a270-1eca15bbc7b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" Oct 30 18:30:21.879406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.879375 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw"] Oct 30 18:30:21.979023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:21.979001 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:22.107536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.107451 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4927ea-a695-4eaa-8443-2e83a61db882-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4927ea-a695-4eaa-8443-2e83a61db882-maas-api-service-ca\") pod \"9c4927ea-a695-4eaa-8443-2e83a61db882\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " Oct 30 18:30:22.107716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.107570 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4927ea-a695-4eaa-8443-2e83a61db882-kube-api-access-hqpbr\" (UniqueName: \"kubernetes.io/projected/9c4927ea-a695-4eaa-8443-2e83a61db882-kube-api-access-hqpbr\") pod \"9c4927ea-a695-4eaa-8443-2e83a61db882\" (UID: \"9c4927ea-a695-4eaa-8443-2e83a61db882\") " Oct 30 18:30:22.107794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.107723 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4927ea-a695-4eaa-8443-2e83a61db882-maas-api-service-ca" pod "9c4927ea-a695-4eaa-8443-2e83a61db882" (UID: "9c4927ea-a695-4eaa-8443-2e83a61db882"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:22.108055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.108038 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4927ea-a695-4eaa-8443-2e83a61db882-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:22.109713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.109693 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4927ea-a695-4eaa-8443-2e83a61db882-kube-api-access-hqpbr" pod "9c4927ea-a695-4eaa-8443-2e83a61db882" (UID: "9c4927ea-a695-4eaa-8443-2e83a61db882"). InnerVolumeSpecName "kube-api-access-hqpbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:22.209333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.209303 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqpbr\" (UniqueName: \"kubernetes.io/projected/9c4927ea-a695-4eaa-8443-2e83a61db882-kube-api-access-hqpbr\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:22.817024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.816987 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" event={"ID":"9c4927ea-a695-4eaa-8443-2e83a61db882","Type":"ContainerDied","Data":"d663ffb96bb7024eeb545eac967953b35bbb7b421df0d522bb81a85e20f55c3b"} Oct 30 18:30:22.817024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.817015 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7" Oct 30 18:30:22.817461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.817082 4582 scope.go:122] "RemoveContainer" containerID="d0b7fce62d79dcd6e94c395cca08d2e56372238349979ba6d3aed9415a04bc51" Oct 30 18:30:22.860446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.860418 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:22.880092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.879399 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7"] Oct 30 18:30:22.891388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.890506 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-200c58-29889750-f4zp7"] Oct 30 18:30:22.978701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:22.978677 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:23.017742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.017713 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-kube-api-access-nzzt4\" (UniqueName: \"kubernetes.io/projected/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-kube-api-access-nzzt4\") pod \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " Oct 30 18:30:23.017955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.017940 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-maas-api-service-ca\") pod \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\" (UID: \"5205b0ec-26f2-49fa-a270-1eca15bbc7b9\") " Oct 30 18:30:23.018342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.018315 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-maas-api-service-ca" pod "5205b0ec-26f2-49fa-a270-1eca15bbc7b9" (UID: "5205b0ec-26f2-49fa-a270-1eca15bbc7b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:23.018446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.018430 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:23.019870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.019835 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-kube-api-access-nzzt4" pod "5205b0ec-26f2-49fa-a270-1eca15bbc7b9" (UID: "5205b0ec-26f2-49fa-a270-1eca15bbc7b9"). InnerVolumeSpecName "kube-api-access-nzzt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:23.119916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.119876 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzt4\" (UniqueName: \"kubernetes.io/projected/5205b0ec-26f2-49fa-a270-1eca15bbc7b9-kube-api-access-nzzt4\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:23.179897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.179858 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b"] Oct 30 18:30:23.180157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.180112 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" containerID="cri-o://f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4" gracePeriod=30 Oct 30 18:30:23.249407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.249367 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx"] Oct 30 18:30:23.249980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.249965 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" containerName="revoke-keys" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.249984 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" containerName="revoke-keys" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250002 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250011 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250019 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250024 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250043 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250047 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250075 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250080 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250090 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250094 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250106 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250111 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250121 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250128 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250243 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250255 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250263 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250271 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250279 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc662d85-9037-48e9-b509-0c2ace6fb267" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250286 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250296 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" containerName="revoke-keys" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250304 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.250687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.250310 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.253675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.253656 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.264090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.264064 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f"] Oct 30 18:30:23.264763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.264741 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.264945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.264765 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.264945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.264811 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.264945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.264838 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" containerName="cleanup" Oct 30 18:30:23.265116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.264998 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" containerName="cleanup" Oct 30 18:30:23.268558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.268539 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.268805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.268789 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-380c08-openshift-default-dockercfg-74pzf\"" Oct 30 18:30:23.282142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.282111 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx"] Oct 30 18:30:23.290432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.290401 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f"] Oct 30 18:30:23.306321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.306287 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4927ea-a695-4eaa-8443-2e83a61db882" path="/var/lib/kubelet/pods/9c4927ea-a695-4eaa-8443-2e83a61db882/volumes" Oct 30 18:30:23.321592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.321561 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.321732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.321612 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.321732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.321638 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctszz\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-kube-api-access-ctszz\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.321879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.321759 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.322535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.321812 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6313580-767d-435f-a756-a2ffe1de64fa-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.322645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322564 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.322645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322628 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.322754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322661 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.322754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322688 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.322850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322748 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znb6t\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-kube-api-access-znb6t\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.322850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322804 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.322927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322877 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07180983-3bdc-4785-a74f-0c1e22b42092-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.322927 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322905 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.322996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322930 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.322996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322944 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.322996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.322966 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.323088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.323000 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6313580-767d-435f-a756-a2ffe1de64fa-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.323088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.323020 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07180983-3bdc-4785-a74f-0c1e22b42092-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.441850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.441727 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.442579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.442556 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6313580-767d-435f-a756-a2ffe1de64fa-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.443556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.443526 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07180983-3bdc-4785-a74f-0c1e22b42092-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.444259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.444238 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.444757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.444737 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.471049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.471019 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctszz\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-kube-api-access-ctszz\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.471301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.471278 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.471465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.471448 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6313580-767d-435f-a756-a2ffe1de64fa-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.471713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.471581 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.471952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.471925 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.472133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.472116 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.472255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.472240 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.472410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.472394 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znb6t\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-kube-api-access-znb6t\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.472541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.472526 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.472660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.472642 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07180983-3bdc-4785-a74f-0c1e22b42092-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.472847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.472809 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.473000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.472983 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.473126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.473110 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.473649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.473628 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.473778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.442465 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.474211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.470318 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.474616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.474596 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.474745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.470402 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.477560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.475487 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.477560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.470951 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07180983-3bdc-4785-a74f-0c1e22b42092-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.477560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.475751 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.477560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.443443 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6313580-767d-435f-a756-a2ffe1de64fa-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.477560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.476067 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.477560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.476498 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.487871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.485865 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07180983-3bdc-4785-a74f-0c1e22b42092-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.487871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.487556 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.492018 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.491992 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6313580-767d-435f-a756-a2ffe1de64fa-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.492893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.492811 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctszz\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-kube-api-access-ctszz\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.495571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.495525 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znb6t\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-kube-api-access-znb6t\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.499578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.499529 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:23.503247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.502841 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.580847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.580079 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-380c08-openshift-default-dockercfg-74pzf\"" Oct 30 18:30:23.589431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.589329 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:23.589878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:23.589812 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:24.059370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.059285 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" event={"ID":"5205b0ec-26f2-49fa-a270-1eca15bbc7b9","Type":"ContainerDied","Data":"be6bd2eaf120f53b0282fda66edebd9f3ff8a407a0738b6ae1ad5b3a2d82d39a"} Oct 30 18:30:24.059370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.059367 4582 scope.go:122] "RemoveContainer" containerID="2fea66928f043455050a51a754d89b553485d3c0f38280054cc7448471792ff3" Oct 30 18:30:24.059994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.059489 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw" Oct 30 18:30:24.480094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.480063 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw"] Oct 30 18:30:24.727872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.624368 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-45dc34-29889750-tqnlw"] Oct 30 18:30:24.727872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.708909 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f"] Oct 30 18:30:24.758891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.758416 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx"] Oct 30 18:30:24.820510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.803093 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:24.820510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.803193 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:24.820510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.803242 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:24.820510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.815027 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:24.820510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.815115 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:24.820510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:24.815185 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:25.124797 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:25.124756 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" event={"ID":"e6313580-767d-435f-a756-a2ffe1de64fa","Type":"ContainerStarted","Data":"8a21a73f9ac877cad0d8b16ab34ca9ed0282f25895e3349bd0db005b5ba77ed8"} Oct 30 18:30:25.131522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:25.131482 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" event={"ID":"07180983-3bdc-4785-a74f-0c1e22b42092","Type":"ContainerStarted","Data":"602c5a745a8fd3492878b1dba97ee8ebb8277c1154f1227d3460a7b70c76b410"} Oct 30 18:30:25.366844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:25.366521 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5205b0ec-26f2-49fa-a270-1eca15bbc7b9" path="/var/lib/kubelet/pods/5205b0ec-26f2-49fa-a270-1eca15bbc7b9/volumes" Oct 30 18:30:25.392206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:25.392159 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:25.723847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:25.716016 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:25.723847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:25.717986 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:26.201659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.199650 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" event={"ID":"e6313580-767d-435f-a756-a2ffe1de64fa","Type":"ContainerStarted","Data":"af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139"} Oct 30 18:30:26.204329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.202864 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" event={"ID":"07180983-3bdc-4785-a74f-0c1e22b42092","Type":"ContainerStarted","Data":"273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658"} Oct 30 18:30:26.263844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.263199 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podStartSLOduration=3.263177203 podStartE2EDuration="3.263177203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:26.249064267 +0000 UTC m=+1923.073458580" watchObservedRunningTime="2026-10-30 18:30:26.263177203 +0000 UTC m=+1923.087571493" Oct 30 18:30:26.303195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.303142 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podStartSLOduration=3.303114193 podStartE2EDuration="3.303114193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:26.299023196 +0000 UTC m=+1923.123417489" watchObservedRunningTime="2026-10-30 18:30:26.303114193 +0000 UTC m=+1923.127508485" Oct 30 18:30:26.592150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.590924 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:26.592415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.592394 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:26.592976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.592940 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 18:30:26.593108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.592996 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 18:30:26.593570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.593550 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 18:30:26.593627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:26.593587 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 18:30:27.335811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:27.335777 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:27.591402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:27.591309 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 18:30:27.591670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:27.591640 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 18:30:27.591895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:27.591309 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 18:30:27.592139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:27.592104 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 18:30:28.521503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.521479 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:30:28.599725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.599283 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.599725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.599340 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 18:30:28.600159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.600131 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.600237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.600182 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 18:30:28.700248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700210 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-podinfo\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700396 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-data\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700478 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-envoy\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700533 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istiod-ca-cert\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700589 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-kube-api-access-gqnw9\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-kube-api-access-gqnw9\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700635 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-certs\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700706 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-socket\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700742 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-token\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-token\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700777 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-credential-socket\") pod \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\" (UID: \"f4510a8c-2c41-45b4-a0fd-8f025c96c857\") " Oct 30 18:30:28.700958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700919 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-certs" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:28.701015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700957 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-socket" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:28.701015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.700976 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istiod-ca-cert" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:28.701127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.701028 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-data" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:28.701216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.701190 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-credential-socket" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:28.701431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.701411 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.701524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.701437 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.701524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.701451 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.701524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.701464 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.701524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.701477 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.702654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.702629 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-podinfo" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:28.703053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.703015 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-token" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:28.703907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.703888 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-kube-api-access-gqnw9" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "kube-api-access-gqnw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:28.704306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.704228 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-envoy" pod "f4510a8c-2c41-45b4-a0fd-8f025c96c857" (UID: "f4510a8c-2c41-45b4-a0fd-8f025c96c857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:28.803035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.802931 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.803035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.802972 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqnw9\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-kube-api-access-gqnw9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.803035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.802989 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.803035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.803003 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4510a8c-2c41-45b4-a0fd-8f025c96c857-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.894844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:28.884296 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:29.226425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.226391 4582 generic.go:356] "Generic (PLEG): container finished" podID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerID="f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4" exitCode=0 Oct 30 18:30:29.226633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.226446 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" Oct 30 18:30:29.226633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.226482 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" event={"ID":"f4510a8c-2c41-45b4-a0fd-8f025c96c857","Type":"ContainerDied","Data":"f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4"} Oct 30 18:30:29.226633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.226549 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b" event={"ID":"f4510a8c-2c41-45b4-a0fd-8f025c96c857","Type":"ContainerDied","Data":"a55962321f6e5bf893f5a4f1448dad223c3fde8b14e7826679ea59dcd533b5b0"} Oct 30 18:30:29.226633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.226580 4582 scope.go:122] "RemoveContainer" containerID="f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4" Oct 30 18:30:29.330840 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.330072 4582 scope.go:122] "RemoveContainer" containerID="f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4" Oct 30 18:30:29.335057 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:29.335013 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4\": container with ID starting with f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4 not found: ID does not exist" containerID="f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4" Oct 30 18:30:29.335337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.335289 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4"} err="failed to get container status \"f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4\": rpc error: code = NotFound desc = could not find container \"f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4\": container with ID starting with f7f6b9d472dc18dc706bae30c69930a78577c7fd900908c480722af84a81ddc4 not found: ID does not exist" Oct 30 18:30:29.414864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.414593 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b"] Oct 30 18:30:29.418158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.418130 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-z7c7b"] Oct 30 18:30:29.596904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.595896 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.598995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.595972 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 18:30:29.598995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.598902 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.598995 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.598943 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 18:30:29.599979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.599940 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:29.708552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.708526 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:29.797798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.797756 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99"] Oct 30 18:30:29.812248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.812130 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-200c58-k7n99"] Oct 30 18:30:29.820698 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:29.820663 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:30.121608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.121581 4582 scope.go:122] "RemoveContainer" containerID="4ae752122768366e6e98a02799c79c0cf41ad83171d71e6c9e45dd6314c68589" Oct 30 18:30:30.131382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.131354 4582 scope.go:122] "RemoveContainer" containerID="bb8b7a969e63764d97d2bb2321df61b5c342fb0b82c0d2bbdcc27171e7335320" Oct 30 18:30:30.140325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.140299 4582 scope.go:122] "RemoveContainer" containerID="f1de7658a4d2933a1b45ee59717bcc979c2d9e5637cb48b2c90659e32dfb064e" Oct 30 18:30:30.260855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.255638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:30.591899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.590251 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.591899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.590313 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 18:30:30.591899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.591058 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.591899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:30.591100 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 18:30:31.302929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:31.302889 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfb9ac7-cf08-461e-a033-16d81ea29094" path="/var/lib/kubelet/pods/1dfb9ac7-cf08-461e-a033-16d81ea29094/volumes" Oct 30 18:30:31.303672 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:31.303653 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" path="/var/lib/kubelet/pods/f4510a8c-2c41-45b4-a0fd-8f025c96c857/volumes" Oct 30 18:30:31.591674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:31.591336 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.591674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:31.591398 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 18:30:31.591674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:31.591438 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.591674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:31.591485 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 18:30:31.880490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:31.880451 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:32.590524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:32.590487 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 18:30:32.591057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:32.590556 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 18:30:33.042716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:33.042685 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:33.043109 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:33.043084 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:33.044007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:33.043984 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 18:30:33.571040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:33.571002 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:33.594534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:33.594507 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:33.595075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:33.595050 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:33.595739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:33.595723 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:30:34.934063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.934032 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh"] Oct 30 18:30:34.934571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.934556 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" Oct 30 18:30:34.934617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.934571 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" Oct 30 18:30:34.934685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.934676 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4510a8c-2c41-45b4-a0fd-8f025c96c857" containerName="istio-proxy" Oct 30 18:30:34.939674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.939656 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:34.952262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.952238 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:30:34.952387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.952290 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:30:34.952387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.952351 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:30:34.952387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.952372 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:30:34.956487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:34.956463 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh"] Oct 30 18:30:35.096765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.096731 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.096944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.096800 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82j8\" (UniqueName: \"kubernetes.io/projected/5fb307a6-8a01-4b64-999c-15986e6c6f82-kube-api-access-b82j8\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.096986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.096940 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.198397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.198314 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.198397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.198389 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82j8\" (UniqueName: \"kubernetes.io/projected/5fb307a6-8a01-4b64-999c-15986e6c6f82-kube-api-access-b82j8\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.198661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.198441 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.199189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.199155 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.200769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.200746 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.215593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.215563 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82j8\" (UniqueName: \"kubernetes.io/projected/5fb307a6-8a01-4b64-999c-15986e6c6f82-kube-api-access-b82j8\") pod \"maas-api-revoke-sub-3daa466a-24mhh\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.252872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.252840 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:35.363125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.363063 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:35.558268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:35.558246 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh"] Oct 30 18:30:35.560859 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:35.560831 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb307a6_8a01_4b64_999c_15986e6c6f82.slice/crio-60f7b9535087c527a93046e50f52cfc18e983c5132bb37449ccfc0fe39d5a330 WatchSource:0}: Error finding container 60f7b9535087c527a93046e50f52cfc18e983c5132bb37449ccfc0fe39d5a330: Status 404 returned error can't find the container with id 60f7b9535087c527a93046e50f52cfc18e983c5132bb37449ccfc0fe39d5a330 Oct 30 18:30:36.225044 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:36.225006 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:30:36.289566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.289525 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" event={"ID":"5fb307a6-8a01-4b64-999c-15986e6c6f82","Type":"ContainerStarted","Data":"d5c5d7cd6d942cb80262e91d2937077a2f55b31e1076e7e7fbdabedc1a71c6af"} Oct 30 18:30:36.289741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.289585 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" event={"ID":"5fb307a6-8a01-4b64-999c-15986e6c6f82","Type":"ContainerStarted","Data":"60f7b9535087c527a93046e50f52cfc18e983c5132bb37449ccfc0fe39d5a330"} Oct 30 18:30:36.326051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.326003 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" podStartSLOduration=2.32598899 podStartE2EDuration="2.32598899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:36.322167139 +0000 UTC m=+1933.146561428" watchObservedRunningTime="2026-10-30 18:30:36.32598899 +0000 UTC m=+1933.150383282" Oct 30 18:30:36.505958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.505885 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:36.612566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.612535 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:36.824891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.809901 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj"] Oct 30 18:30:36.824891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.817213 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:36.857713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.857678 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj"] Oct 30 18:30:36.920963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.920909 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825zc\" (UniqueName: \"kubernetes.io/projected/3e910e0c-b792-4da8-9f18-6e53081c8371-kube-api-access-825zc\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:36.921139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.921035 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:36.921207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.921141 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:36.948971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.948927 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv"] Oct 30 18:30:36.949189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.949158 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" containerID="cri-o://8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548" gracePeriod=30 Oct 30 18:30:36.980056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:36.980029 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:37.022219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.022196 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:37.022310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.022266 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:37.022416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.022405 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825zc\" (UniqueName: \"kubernetes.io/projected/3e910e0c-b792-4da8-9f18-6e53081c8371-kube-api-access-825zc\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:37.022841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.022803 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:37.024595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.024569 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:37.038690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.038670 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-825zc\" (UniqueName: \"kubernetes.io/projected/3e910e0c-b792-4da8-9f18-6e53081c8371-kube-api-access-825zc\") pod \"maas-api-revoke-sub-c8cd7bca-rnzsj\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:37.137000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.136968 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:37.298592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.298555 4582 generic.go:356] "Generic (PLEG): container finished" podID="5fb307a6-8a01-4b64-999c-15986e6c6f82" containerID="d5c5d7cd6d942cb80262e91d2937077a2f55b31e1076e7e7fbdabedc1a71c6af" exitCode=0 Oct 30 18:30:37.300566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.300540 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" event={"ID":"5fb307a6-8a01-4b64-999c-15986e6c6f82","Type":"ContainerDied","Data":"d5c5d7cd6d942cb80262e91d2937077a2f55b31e1076e7e7fbdabedc1a71c6af"} Oct 30 18:30:37.316578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.316552 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:37.438780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:37.438746 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj"] Oct 30 18:30:37.441625 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:37.441599 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e910e0c_b792_4da8_9f18_6e53081c8371.slice/crio-ff46fbdf94129a8ed078cd9cdb57d11c7fe927a10689611e441a37231232c0b5 WatchSource:0}: Error finding container ff46fbdf94129a8ed078cd9cdb57d11c7fe927a10689611e441a37231232c0b5: Status 404 returned error can't find the container with id ff46fbdf94129a8ed078cd9cdb57d11c7fe927a10689611e441a37231232c0b5 Oct 30 18:30:38.291844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.290807 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b"] Oct 30 18:30:38.322021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.321988 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.380756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.380630 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" event={"ID":"3e910e0c-b792-4da8-9f18-6e53081c8371","Type":"ContainerStarted","Data":"087629a5c07c9969e498883fd64d06d5881c936534bb880c224f858cf05fc93f"} Oct 30 18:30:38.380756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.380692 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" event={"ID":"3e910e0c-b792-4da8-9f18-6e53081c8371","Type":"ContainerStarted","Data":"ff46fbdf94129a8ed078cd9cdb57d11c7fe927a10689611e441a37231232c0b5"} Oct 30 18:30:38.417895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.407871 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg"] Oct 30 18:30:38.436327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.436302 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.495963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.484642 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541087 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541222 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541252 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541306 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2424245e-41e9-40a7-bc54-73467dbd9949-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541345 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541376 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2s6j\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-kube-api-access-f2s6j\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541456 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2424245e-41e9-40a7-bc54-73467dbd9949-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.555692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.541507 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.645974 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646040 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646076 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646104 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646151 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa979efc-88a6-433f-897e-997eb4c3b11e-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646178 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646239 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646288 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646340 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646367 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646416 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2424245e-41e9-40a7-bc54-73467dbd9949-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646448 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k9s\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-kube-api-access-p4k9s\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646482 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646515 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2s6j\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-kube-api-access-f2s6j\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646553 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646597 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646631 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2424245e-41e9-40a7-bc54-73467dbd9949-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.646676 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.648844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.647189 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.650184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.649314 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.650184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.649566 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.683088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.681354 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.701535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.687561 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2424245e-41e9-40a7-bc54-73467dbd9949-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.701535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.687620 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.748897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.748866 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b"] Oct 30 18:30:38.750229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.749839 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k9s\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-kube-api-access-p4k9s\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.750854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.750834 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.751275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.751254 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.752066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.752044 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.752518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.752498 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.753051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.753032 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.753507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.753489 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa979efc-88a6-433f-897e-997eb4c3b11e-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.753901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.753879 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.754070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.754051 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.763575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.754463 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.787566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.753423 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.787566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.770140 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.787566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.751842 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.787566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.780163 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa979efc-88a6-433f-897e-997eb4c3b11e-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.836003 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.834940 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg"] Oct 30 18:30:38.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.872358 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.879318 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.879732 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.909155 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.916693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.916499 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2424245e-41e9-40a7-bc54-73467dbd9949-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.918725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.918523 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2s6j\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-kube-api-access-f2s6j\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:38.918725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.918688 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k9s\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-kube-api-access-p4k9s\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:38.957666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:38.957592 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" podStartSLOduration=2.957570712 podStartE2EDuration="2.957570712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:38.432612052 +0000 UTC m=+1935.257006360" watchObservedRunningTime="2026-10-30 18:30:38.957570712 +0000 UTC m=+1935.781965005" Oct 30 18:30:39.108340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.108031 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:39.115082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.115045 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:39.646843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.641287 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:39.694800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.694010 4582 generic.go:356] "Generic (PLEG): container finished" podID="3e910e0c-b792-4da8-9f18-6e53081c8371" containerID="087629a5c07c9969e498883fd64d06d5881c936534bb880c224f858cf05fc93f" exitCode=0 Oct 30 18:30:39.694800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.694161 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" event={"ID":"3e910e0c-b792-4da8-9f18-6e53081c8371","Type":"ContainerDied","Data":"087629a5c07c9969e498883fd64d06d5881c936534bb880c224f858cf05fc93f"} Oct 30 18:30:39.720839 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.717348 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:39.754897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.749690 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb307a6-8a01-4b64-999c-15986e6c6f82-kube-api-access-b82j8\" (UniqueName: \"kubernetes.io/projected/5fb307a6-8a01-4b64-999c-15986e6c6f82-kube-api-access-b82j8\") pod \"5fb307a6-8a01-4b64-999c-15986e6c6f82\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " Oct 30 18:30:39.754897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.752416 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb307a6-8a01-4b64-999c-15986e6c6f82-kube-api-access-b82j8" pod "5fb307a6-8a01-4b64-999c-15986e6c6f82" (UID: "5fb307a6-8a01-4b64-999c-15986e6c6f82"). InnerVolumeSpecName "kube-api-access-b82j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:39.861040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.852968 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-xks-ca\") pod \"5fb307a6-8a01-4b64-999c-15986e6c6f82\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " Oct 30 18:30:39.861040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.853670 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-openshift-ca\") pod \"5fb307a6-8a01-4b64-999c-15986e6c6f82\" (UID: \"5fb307a6-8a01-4b64-999c-15986e6c6f82\") " Oct 30 18:30:39.861040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.854508 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-openshift-ca" pod "5fb307a6-8a01-4b64-999c-15986e6c6f82" (UID: "5fb307a6-8a01-4b64-999c-15986e6c6f82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:39.861040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.854897 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:39.861040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.854934 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82j8\" (UniqueName: \"kubernetes.io/projected/5fb307a6-8a01-4b64-999c-15986e6c6f82-kube-api-access-b82j8\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:39.861468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.861418 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-xks-ca" pod "5fb307a6-8a01-4b64-999c-15986e6c6f82" (UID: "5fb307a6-8a01-4b64-999c-15986e6c6f82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:39.956065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:39.956029 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb307a6-8a01-4b64-999c-15986e6c6f82-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:40.012292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.007140 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b"] Oct 30 18:30:40.014296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.014255 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:40.014398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.014349 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:40.014441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.014395 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:40.082417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.082310 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg"] Oct 30 18:30:40.085088 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:40.085064 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa979efc_88a6_433f_897e_997eb4c3b11e.slice/crio-e29de418f3538ef3c51fa9c578c658c5e5a753b721b8cd376b7c95dc03526b8b WatchSource:0}: Error finding container e29de418f3538ef3c51fa9c578c658c5e5a753b721b8cd376b7c95dc03526b8b: Status 404 returned error can't find the container with id e29de418f3538ef3c51fa9c578c658c5e5a753b721b8cd376b7c95dc03526b8b Oct 30 18:30:40.087941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.087908 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:40.088030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.087976 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:40.088030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.088010 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:30:40.703711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.703679 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" event={"ID":"2424245e-41e9-40a7-bc54-73467dbd9949","Type":"ContainerStarted","Data":"35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27"} Oct 30 18:30:40.704213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.703733 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" event={"ID":"2424245e-41e9-40a7-bc54-73467dbd9949","Type":"ContainerStarted","Data":"71c5d98ddedf6e589a5f4c9efff421244a960a95c45ef8754b5c25e21e8ef69d"} Oct 30 18:30:40.705358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.705340 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" Oct 30 18:30:40.705493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.705349 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh" event={"ID":"5fb307a6-8a01-4b64-999c-15986e6c6f82","Type":"ContainerDied","Data":"60f7b9535087c527a93046e50f52cfc18e983c5132bb37449ccfc0fe39d5a330"} Oct 30 18:30:40.705493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.705392 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f7b9535087c527a93046e50f52cfc18e983c5132bb37449ccfc0fe39d5a330" Oct 30 18:30:40.707179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.707148 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" event={"ID":"aa979efc-88a6-433f-897e-997eb4c3b11e","Type":"ContainerStarted","Data":"644e2da1447c05f27ac8be3db9980ac0ec53f1ffd0f372e08d8701c2878d89db"} Oct 30 18:30:40.707293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.707189 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" event={"ID":"aa979efc-88a6-433f-897e-997eb4c3b11e","Type":"ContainerStarted","Data":"e29de418f3538ef3c51fa9c578c658c5e5a753b721b8cd376b7c95dc03526b8b"} Oct 30 18:30:40.760215 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.760154 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podStartSLOduration=2.760138181 podStartE2EDuration="2.760138181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:40.749527947 +0000 UTC m=+1937.573922239" watchObservedRunningTime="2026-10-30 18:30:40.760138181 +0000 UTC m=+1937.584532472" Oct 30 18:30:40.810221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.810146 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podStartSLOduration=2.810127531 podStartE2EDuration="2.810127531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:40.808763384 +0000 UTC m=+1937.633157681" watchObservedRunningTime="2026-10-30 18:30:40.810127531 +0000 UTC m=+1937.634521823" Oct 30 18:30:40.895200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.895177 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:40.974814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.974773 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-xks-ca\") pod \"3e910e0c-b792-4da8-9f18-6e53081c8371\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " Oct 30 18:30:40.975026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.974847 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e910e0c-b792-4da8-9f18-6e53081c8371-kube-api-access-825zc\" (UniqueName: \"kubernetes.io/projected/3e910e0c-b792-4da8-9f18-6e53081c8371-kube-api-access-825zc\") pod \"3e910e0c-b792-4da8-9f18-6e53081c8371\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " Oct 30 18:30:40.975026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.974991 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-openshift-ca\") pod \"3e910e0c-b792-4da8-9f18-6e53081c8371\" (UID: \"3e910e0c-b792-4da8-9f18-6e53081c8371\") " Oct 30 18:30:40.975358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.975332 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-openshift-ca" pod "3e910e0c-b792-4da8-9f18-6e53081c8371" (UID: "3e910e0c-b792-4da8-9f18-6e53081c8371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:40.975575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.975549 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:40.976976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.976955 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e910e0c-b792-4da8-9f18-6e53081c8371-kube-api-access-825zc" pod "3e910e0c-b792-4da8-9f18-6e53081c8371" (UID: "3e910e0c-b792-4da8-9f18-6e53081c8371"). InnerVolumeSpecName "kube-api-access-825zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:40.977393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:40.977374 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-xks-ca" pod "3e910e0c-b792-4da8-9f18-6e53081c8371" (UID: "3e910e0c-b792-4da8-9f18-6e53081c8371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:41.076879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.076835 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e910e0c-b792-4da8-9f18-6e53081c8371-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:41.076879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.076865 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-825zc\" (UniqueName: \"kubernetes.io/projected/3e910e0c-b792-4da8-9f18-6e53081c8371-kube-api-access-825zc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:41.111358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.109667 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:41.111564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.111377 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:41.111564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.111448 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 18:30:41.116090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.116070 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:41.117166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.117146 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:30:41.117219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.117186 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:30:41.419959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.419919 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:41.717385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.717355 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" Oct 30 18:30:41.717777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.717758 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj" event={"ID":"3e910e0c-b792-4da8-9f18-6e53081c8371","Type":"ContainerDied","Data":"ff46fbdf94129a8ed078cd9cdb57d11c7fe927a10689611e441a37231232c0b5"} Oct 30 18:30:41.717875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:41.717792 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff46fbdf94129a8ed078cd9cdb57d11c7fe927a10689611e441a37231232c0b5" Oct 30 18:30:42.109636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.109604 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:42.109774 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.109653 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 18:30:42.116381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.116354 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:30:42.116507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.116411 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:30:42.328606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.328581 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:30:42.405218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405181 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-data\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405221 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-token\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-token\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405299 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-socket\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405335 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-certs\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405373 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-podinfo\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405399 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-envoy\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405433 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istiod-ca-cert\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405614 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-kube-api-access-z77sr\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-kube-api-access-z77sr\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405641 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-credential-socket\") pod \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\" (UID: \"94fae2ec-2b00-45dd-b9ef-06decebd68b9\") " Oct 30 18:30:42.405813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405634 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-socket" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:42.406025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405911 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-data" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:42.406025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405959 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-certs" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:42.406025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.405997 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istiod-ca-cert" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:42.406248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.406082 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-credential-socket" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:42.406407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.406331 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.406407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.406355 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.406407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.406367 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.406407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.406376 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.406407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.406387 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.407783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.407764 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-envoy" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:42.408315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.408293 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-kube-api-access-z77sr" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "kube-api-access-z77sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:42.408658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.408641 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-podinfo" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:42.408736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.408723 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-token" pod "94fae2ec-2b00-45dd-b9ef-06decebd68b9" (UID: "94fae2ec-2b00-45dd-b9ef-06decebd68b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:42.507391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.507356 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z77sr\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-kube-api-access-z77sr\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.507391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.507386 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.507391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.507396 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.507623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.507406 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94fae2ec-2b00-45dd-b9ef-06decebd68b9-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:42.725592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.725557 4582 generic.go:356] "Generic (PLEG): container finished" podID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerID="8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548" exitCode=0 Oct 30 18:30:42.725996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.725608 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" Oct 30 18:30:42.725996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.725647 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" event={"ID":"94fae2ec-2b00-45dd-b9ef-06decebd68b9","Type":"ContainerDied","Data":"8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548"} Oct 30 18:30:42.725996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.725700 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv" event={"ID":"94fae2ec-2b00-45dd-b9ef-06decebd68b9","Type":"ContainerDied","Data":"b2a48696d27e4675cc96c5aa0fbb0a6cd86c260bbf08d6dd26ea7dc3ff5986e8"} Oct 30 18:30:42.725996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.725730 4582 scope.go:122] "RemoveContainer" containerID="8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548" Oct 30 18:30:42.740049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.740031 4582 scope.go:122] "RemoveContainer" containerID="63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2" Oct 30 18:30:42.798843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.798757 4582 scope.go:122] "RemoveContainer" containerID="8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548" Oct 30 18:30:42.801604 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:42.801235 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548\": container with ID starting with 8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548 not found: ID does not exist" containerID="8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548" Oct 30 18:30:42.801604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.801278 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548"} err="failed to get container status \"8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548\": rpc error: code = NotFound desc = could not find container \"8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548\": container with ID starting with 8d8c6e657df731dcd2b6639eff3de342bc3c493f3c166dcd4c27e936f8f56548 not found: ID does not exist" Oct 30 18:30:42.801604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.801306 4582 scope.go:122] "RemoveContainer" containerID="63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2" Oct 30 18:30:42.802103 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:30:42.802039 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2\": container with ID starting with 63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2 not found: ID does not exist" containerID="63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2" Oct 30 18:30:42.802103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.802075 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2"} err="failed to get container status \"63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2\": rpc error: code = NotFound desc = could not find container \"63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2\": container with ID starting with 63a862bc238c4ad31d11b2bacd8718016051c002684ec513a20bc3062ba703c2 not found: ID does not exist" Oct 30 18:30:42.803932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.803896 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv"] Oct 30 18:30:42.807947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:42.807618 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-200c58-openshift-default-75b6749c97-9cbrv"] Oct 30 18:30:43.109863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:43.109463 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:43.109863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:43.109515 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 18:30:43.115965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:43.115938 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:30:43.116044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:43.115980 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:30:43.312517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:43.312477 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" path="/var/lib/kubelet/pods/94fae2ec-2b00-45dd-b9ef-06decebd68b9/volumes" Oct 30 18:30:43.501375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:43.501339 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:44.101766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.101727 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89"] Oct 30 18:30:44.102326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102309 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3e910e0c-b792-4da8-9f18-6e53081c8371" containerName="revoke-keys" Oct 30 18:30:44.102373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102327 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e910e0c-b792-4da8-9f18-6e53081c8371" containerName="revoke-keys" Oct 30 18:30:44.102373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102353 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5fb307a6-8a01-4b64-999c-15986e6c6f82" containerName="revoke-keys" Oct 30 18:30:44.102373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102361 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb307a6-8a01-4b64-999c-15986e6c6f82" containerName="revoke-keys" Oct 30 18:30:44.102491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102378 4582 cpu_manager.go:414] "containerMap: removing container" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" Oct 30 18:30:44.102491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102383 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" Oct 30 18:30:44.102491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102391 4582 cpu_manager.go:414] "containerMap: removing container" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" Oct 30 18:30:44.102491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102396 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" Oct 30 18:30:44.102635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102521 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e910e0c-b792-4da8-9f18-6e53081c8371" containerName="revoke-keys" Oct 30 18:30:44.102635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102531 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb307a6-8a01-4b64-999c-15986e6c6f82" containerName="revoke-keys" Oct 30 18:30:44.102635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102540 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" Oct 30 18:30:44.102635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.102545 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fae2ec-2b00-45dd-b9ef-06decebd68b9" containerName="istio-proxy" Oct 30 18:30:44.107718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.107700 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.109293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.109270 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:44.109367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.109323 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 18:30:44.115569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.115551 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:30:44.115654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.115593 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:30:44.122357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.122227 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89"] Oct 30 18:30:44.128080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.128060 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:30:44.128194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.128179 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:30:44.128282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.128243 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:30:44.129224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.129204 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:30:44.130624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.130600 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.130716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.130649 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.130784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.130766 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzm2t\" (UniqueName: \"kubernetes.io/projected/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-kube-api-access-wzm2t\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.232343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.232306 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzm2t\" (UniqueName: \"kubernetes.io/projected/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-kube-api-access-wzm2t\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.232542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.232431 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.232542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.232463 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.233250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.233224 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.235880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.235851 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.295049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.294520 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzm2t\" (UniqueName: \"kubernetes.io/projected/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-kube-api-access-wzm2t\") pod \"maas-api-revoke-sub-5978fbab-hxj89\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.421097 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.421059 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:44.802213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:44.802163 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89"] Oct 30 18:30:45.109017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.108940 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:45.109017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.108992 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 18:30:45.116273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.116249 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:30:45.116361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.116290 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:30:45.550845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.547194 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:45.800898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.800782 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" event={"ID":"68b4fda4-5ce4-46db-bb11-2cefb6520a4e","Type":"ContainerStarted","Data":"ae6969277383b3e39f83ad9c56aa77aee24f638fc75cf6316982ae7aa223b1b4"} Oct 30 18:30:45.800898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.800868 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" event={"ID":"68b4fda4-5ce4-46db-bb11-2cefb6520a4e","Type":"ContainerStarted","Data":"d4a48bc089be95429b4f18a104fa0899e5cd1600614599026b8916fc3034cdd6"} Oct 30 18:30:45.834899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:45.834845 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" podStartSLOduration=1.834182362 podStartE2EDuration="1.834182362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:45.833001568 +0000 UTC m=+1942.657395860" watchObservedRunningTime="2026-10-30 18:30:45.834182362 +0000 UTC m=+1942.658576656" Oct 30 18:30:46.109418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:46.109340 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:46.109418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:46.109394 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 18:30:46.115696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:46.115670 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:30:46.115868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:46.115844 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:30:46.812042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:46.812008 4582 generic.go:356] "Generic (PLEG): container finished" podID="68b4fda4-5ce4-46db-bb11-2cefb6520a4e" containerID="ae6969277383b3e39f83ad9c56aa77aee24f638fc75cf6316982ae7aa223b1b4" exitCode=0 Oct 30 18:30:46.812236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:46.812090 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" event={"ID":"68b4fda4-5ce4-46db-bb11-2cefb6520a4e","Type":"ContainerDied","Data":"ae6969277383b3e39f83ad9c56aa77aee24f638fc75cf6316982ae7aa223b1b4"} Oct 30 18:30:47.109013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:47.108931 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 18:30:47.109013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:47.108986 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 18:30:47.116499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:47.116470 4582 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:30:47.116867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:47.116530 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:30:47.700224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:47.700172 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:47.801757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:47.801715 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:48.003490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.003462 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:48.113208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.113180 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:48.113576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.113552 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:48.114309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.114289 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:30:48.120721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.120699 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:48.121136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.121125 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:48.121841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.121810 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:30:48.140802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.140782 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-kube-api-access-wzm2t\" (UniqueName: \"kubernetes.io/projected/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-kube-api-access-wzm2t\") pod \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " Oct 30 18:30:48.140954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.140938 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-xks-ca\") pod \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " Oct 30 18:30:48.141083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.141068 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-openshift-ca\") pod \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\" (UID: \"68b4fda4-5ce4-46db-bb11-2cefb6520a4e\") " Oct 30 18:30:48.141470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.141439 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-openshift-ca" pod "68b4fda4-5ce4-46db-bb11-2cefb6520a4e" (UID: "68b4fda4-5ce4-46db-bb11-2cefb6520a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:48.141653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.141632 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:48.142898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.142868 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-kube-api-access-wzm2t" pod "68b4fda4-5ce4-46db-bb11-2cefb6520a4e" (UID: "68b4fda4-5ce4-46db-bb11-2cefb6520a4e"). InnerVolumeSpecName "kube-api-access-wzm2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:48.143020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.142999 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-xks-ca" pod "68b4fda4-5ce4-46db-bb11-2cefb6520a4e" (UID: "68b4fda4-5ce4-46db-bb11-2cefb6520a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:48.241032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.240929 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc"] Oct 30 18:30:48.241861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.241796 4582 cpu_manager.go:414] "containerMap: removing container" podUID="68b4fda4-5ce4-46db-bb11-2cefb6520a4e" containerName="revoke-keys" Oct 30 18:30:48.241861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.241838 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b4fda4-5ce4-46db-bb11-2cefb6520a4e" containerName="revoke-keys" Oct 30 18:30:48.242149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.242044 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b4fda4-5ce4-46db-bb11-2cefb6520a4e" containerName="revoke-keys" Oct 30 18:30:48.245604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.244946 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzm2t\" (UniqueName: \"kubernetes.io/projected/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-kube-api-access-wzm2t\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:48.245604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.244974 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68b4fda4-5ce4-46db-bb11-2cefb6520a4e-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:48.246506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.246484 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.270329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.270297 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc"] Oct 30 18:30:48.307428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.307394 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:48.346477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.346445 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszt7\" (UniqueName: \"kubernetes.io/projected/fc094205-a88e-42fe-83db-3445bed8c279-kube-api-access-wszt7\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.346647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.346628 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.346783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.346767 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.448599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.448562 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.448812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.448641 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszt7\" (UniqueName: \"kubernetes.io/projected/fc094205-a88e-42fe-83db-3445bed8c279-kube-api-access-wszt7\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.448812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.448740 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.449451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.449427 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.451161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.451136 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.470339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.470307 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszt7\" (UniqueName: \"kubernetes.io/projected/fc094205-a88e-42fe-83db-3445bed8c279-kube-api-access-wszt7\") pod \"maas-api-revoke-sub-f6db7832-ncbsc\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.571129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.571050 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:48.587210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.587183 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:48.831487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.831410 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" Oct 30 18:30:48.831487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.831448 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89" event={"ID":"68b4fda4-5ce4-46db-bb11-2cefb6520a4e","Type":"ContainerDied","Data":"d4a48bc089be95429b4f18a104fa0899e5cd1600614599026b8916fc3034cdd6"} Oct 30 18:30:48.831742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.831501 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4a48bc089be95429b4f18a104fa0899e5cd1600614599026b8916fc3034cdd6" Oct 30 18:30:48.873921 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:48.873760 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc"] Oct 30 18:30:48.876398 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:48.876362 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc094205_a88e_42fe_83db_3445bed8c279.slice/crio-bdb551b2e84e8d87764b7c2fe7d8772cf0cecbcfbc3256f76c79ea8ae2580e38 WatchSource:0}: Error finding container bdb551b2e84e8d87764b7c2fe7d8772cf0cecbcfbc3256f76c79ea8ae2580e38: Status 404 returned error can't find the container with id bdb551b2e84e8d87764b7c2fe7d8772cf0cecbcfbc3256f76c79ea8ae2580e38 Oct 30 18:30:49.678790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:49.678753 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:49.840317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:49.840280 4582 generic.go:356] "Generic (PLEG): container finished" podID="fc094205-a88e-42fe-83db-3445bed8c279" containerID="d5c5e23b2016bbf683fa43ec8941ff72d8620199ae689adb90e5bd3b359983b1" exitCode=0 Oct 30 18:30:49.840523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:49.840365 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" event={"ID":"fc094205-a88e-42fe-83db-3445bed8c279","Type":"ContainerDied","Data":"d5c5e23b2016bbf683fa43ec8941ff72d8620199ae689adb90e5bd3b359983b1"} Oct 30 18:30:49.840523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:49.840423 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" event={"ID":"fc094205-a88e-42fe-83db-3445bed8c279","Type":"ContainerStarted","Data":"bdb551b2e84e8d87764b7c2fe7d8772cf0cecbcfbc3256f76c79ea8ae2580e38"} Oct 30 18:30:51.009195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.009168 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:51.182382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.182350 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc094205-a88e-42fe-83db-3445bed8c279-kube-api-access-wszt7\" (UniqueName: \"kubernetes.io/projected/fc094205-a88e-42fe-83db-3445bed8c279-kube-api-access-wszt7\") pod \"fc094205-a88e-42fe-83db-3445bed8c279\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " Oct 30 18:30:51.182570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.182459 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-xks-ca\") pod \"fc094205-a88e-42fe-83db-3445bed8c279\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " Oct 30 18:30:51.182570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.182533 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-openshift-ca\") pod \"fc094205-a88e-42fe-83db-3445bed8c279\" (UID: \"fc094205-a88e-42fe-83db-3445bed8c279\") " Oct 30 18:30:51.182942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.182905 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-openshift-ca" pod "fc094205-a88e-42fe-83db-3445bed8c279" (UID: "fc094205-a88e-42fe-83db-3445bed8c279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:51.183170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.183150 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:51.184552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.184523 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc094205-a88e-42fe-83db-3445bed8c279-kube-api-access-wszt7" pod "fc094205-a88e-42fe-83db-3445bed8c279" (UID: "fc094205-a88e-42fe-83db-3445bed8c279"). InnerVolumeSpecName "kube-api-access-wszt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:51.184665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.184586 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-xks-ca" pod "fc094205-a88e-42fe-83db-3445bed8c279" (UID: "fc094205-a88e-42fe-83db-3445bed8c279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:51.284833 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.284785 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wszt7\" (UniqueName: \"kubernetes.io/projected/fc094205-a88e-42fe-83db-3445bed8c279-kube-api-access-wszt7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:51.284997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.284848 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc094205-a88e-42fe-83db-3445bed8c279-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:51.470713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.470633 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:51.855553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.855469 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" Oct 30 18:30:51.855553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.855510 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc" event={"ID":"fc094205-a88e-42fe-83db-3445bed8c279","Type":"ContainerDied","Data":"bdb551b2e84e8d87764b7c2fe7d8772cf0cecbcfbc3256f76c79ea8ae2580e38"} Oct 30 18:30:51.855553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.855555 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb551b2e84e8d87764b7c2fe7d8772cf0cecbcfbc3256f76c79ea8ae2580e38" Oct 30 18:30:51.945648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:51.945594 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:52.084246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:52.084210 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:53.359068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.359021 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:53.644862 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.644830 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:53.743524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.743493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:53.958378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.956386 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42"] Oct 30 18:30:53.958378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.957506 4582 cpu_manager.go:414] "containerMap: removing container" podUID="fc094205-a88e-42fe-83db-3445bed8c279" containerName="revoke-keys" Oct 30 18:30:53.958378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.957529 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc094205-a88e-42fe-83db-3445bed8c279" containerName="revoke-keys" Oct 30 18:30:53.958378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.957733 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc094205-a88e-42fe-83db-3445bed8c279" containerName="revoke-keys" Oct 30 18:30:53.966273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.966247 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:53.983296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.982522 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ab244c\"" Oct 30 18:30:53.984304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.984271 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42"] Oct 30 18:30:53.995805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:53.995735 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ab244c-dockercfg-h5z7n\"" Oct 30 18:30:54.032397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.032365 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b02e285-5f9b-4702-b7a1-666dec453d4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.033970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.032593 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxxd\" (UniqueName: \"kubernetes.io/projected/5b02e285-5f9b-4702-b7a1-666dec453d4b-kube-api-access-8qxxd\") pod \"payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.133520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.133486 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxxd\" (UniqueName: \"kubernetes.io/projected/5b02e285-5f9b-4702-b7a1-666dec453d4b-kube-api-access-8qxxd\") pod \"payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.133677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.133534 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b02e285-5f9b-4702-b7a1-666dec453d4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.134266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.134247 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b02e285-5f9b-4702-b7a1-666dec453d4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.150200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.150176 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxxd\" (UniqueName: \"kubernetes.io/projected/5b02e285-5f9b-4702-b7a1-666dec453d4b-kube-api-access-8qxxd\") pod \"payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.300024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.299944 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.597054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.597026 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42"] Oct 30 18:30:54.599801 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:54.599772 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b02e285_5f9b_4702_b7a1_666dec453d4b.slice/crio-361d654af294c09ca6e28b79a34bb090e42a6ba26e69d7bfe751bc033eeddaf3 WatchSource:0}: Error finding container 361d654af294c09ca6e28b79a34bb090e42a6ba26e69d7bfe751bc033eeddaf3: Status 404 returned error can't find the container with id 361d654af294c09ca6e28b79a34bb090e42a6ba26e69d7bfe751bc033eeddaf3 Oct 30 18:30:54.878790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.878753 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" event={"ID":"5b02e285-5f9b-4702-b7a1-666dec453d4b","Type":"ContainerStarted","Data":"a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51"} Oct 30 18:30:54.879026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.878961 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:30:54.879026 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.878983 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" event={"ID":"5b02e285-5f9b-4702-b7a1-666dec453d4b","Type":"ContainerStarted","Data":"361d654af294c09ca6e28b79a34bb090e42a6ba26e69d7bfe751bc033eeddaf3"} Oct 30 18:30:54.916429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:54.916374 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" podStartSLOduration=1.916360864 podStartE2EDuration="1.916360864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:54.913447098 +0000 UTC m=+1951.737841425" watchObservedRunningTime="2026-10-30 18:30:54.916360864 +0000 UTC m=+1951.740755163" Oct 30 18:30:55.201228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.201152 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:55.962578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.962543 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r"] Oct 30 18:30:55.967691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.967666 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:55.984460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.984422 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r"] Oct 30 18:30:55.989203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.989180 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:30:55.989203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.989194 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:30:55.989487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.989468 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:30:55.989552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:55.989498 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:30:56.059325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.059299 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.059483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.059412 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcr8\" (UniqueName: \"kubernetes.io/projected/ceb28498-fa64-4247-a556-e8d79a41c5d8-kube-api-access-8fcr8\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.059548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.059490 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.160898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.160864 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.161079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.161018 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.161148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.161136 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcr8\" (UniqueName: \"kubernetes.io/projected/ceb28498-fa64-4247-a556-e8d79a41c5d8-kube-api-access-8fcr8\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.161625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.161604 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.163322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.163298 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.177790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.177765 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcr8\" (UniqueName: \"kubernetes.io/projected/ceb28498-fa64-4247-a556-e8d79a41c5d8-kube-api-access-8fcr8\") pod \"maas-api-revoke-sub-44c2477f-whc8r\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.282881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.282780 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:56.578665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.578612 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r"] Oct 30 18:30:56.581464 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:56.581437 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb28498_fa64_4247_a556_e8d79a41c5d8.slice/crio-1781a999512db264239c1e1fa12f8e7a897496107fc267325e17623b39e55320 WatchSource:0}: Error finding container 1781a999512db264239c1e1fa12f8e7a897496107fc267325e17623b39e55320: Status 404 returned error can't find the container with id 1781a999512db264239c1e1fa12f8e7a897496107fc267325e17623b39e55320 Oct 30 18:30:56.896771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.896739 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" event={"ID":"ceb28498-fa64-4247-a556-e8d79a41c5d8","Type":"ContainerStarted","Data":"167a7cffea4ef28acb290871086487609af1f93c37bd25468ed97bb151d3d9aa"} Oct 30 18:30:56.896958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.896790 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" event={"ID":"ceb28498-fa64-4247-a556-e8d79a41c5d8","Type":"ContainerStarted","Data":"1781a999512db264239c1e1fa12f8e7a897496107fc267325e17623b39e55320"} Oct 30 18:30:56.932486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.932352 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" podStartSLOduration=1.932334525 podStartE2EDuration="1.932334525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:56.929322452 +0000 UTC m=+1953.753716744" watchObservedRunningTime="2026-10-30 18:30:56.932334525 +0000 UTC m=+1953.756728832" Oct 30 18:30:56.942151 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:56.942122 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:57.905834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:57.905793 4582 generic.go:356] "Generic (PLEG): container finished" podID="ceb28498-fa64-4247-a556-e8d79a41c5d8" containerID="167a7cffea4ef28acb290871086487609af1f93c37bd25468ed97bb151d3d9aa" exitCode=0 Oct 30 18:30:57.906235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:57.905848 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" event={"ID":"ceb28498-fa64-4247-a556-e8d79a41c5d8","Type":"ContainerDied","Data":"167a7cffea4ef28acb290871086487609af1f93c37bd25468ed97bb151d3d9aa"} Oct 30 18:30:58.593281 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.593249 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc"] Oct 30 18:30:58.597486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.597469 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.614903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.614871 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc"] Oct 30 18:30:58.688607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.688573 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.688752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.688649 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.688752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.688722 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxfc\" (UniqueName: \"kubernetes.io/projected/32086a5d-e428-4c95-be26-8aa59580733d-kube-api-access-gzxfc\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.762480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.762447 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:58.789484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.789455 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.789661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.789523 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.789661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.789561 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxfc\" (UniqueName: \"kubernetes.io/projected/32086a5d-e428-4c95-be26-8aa59580733d-kube-api-access-gzxfc\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.790269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.790242 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.792036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.792016 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.806052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.806028 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxfc\" (UniqueName: \"kubernetes.io/projected/32086a5d-e428-4c95-be26-8aa59580733d-kube-api-access-gzxfc\") pod \"maas-api-revoke-sub-89197c7f-z5mmc\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:58.910772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:58.910739 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:30:59.097147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.097124 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:59.193771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.193706 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-openshift-ca\") pod \"ceb28498-fa64-4247-a556-e8d79a41c5d8\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " Oct 30 18:30:59.193939 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.193851 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-xks-ca\") pod \"ceb28498-fa64-4247-a556-e8d79a41c5d8\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " Oct 30 18:30:59.193939 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.193888 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb28498-fa64-4247-a556-e8d79a41c5d8-kube-api-access-8fcr8\" (UniqueName: \"kubernetes.io/projected/ceb28498-fa64-4247-a556-e8d79a41c5d8-kube-api-access-8fcr8\") pod \"ceb28498-fa64-4247-a556-e8d79a41c5d8\" (UID: \"ceb28498-fa64-4247-a556-e8d79a41c5d8\") " Oct 30 18:30:59.194068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.194023 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-openshift-ca" pod "ceb28498-fa64-4247-a556-e8d79a41c5d8" (UID: "ceb28498-fa64-4247-a556-e8d79a41c5d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:59.194296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.194280 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:59.195893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.195872 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-xks-ca" pod "ceb28498-fa64-4247-a556-e8d79a41c5d8" (UID: "ceb28498-fa64-4247-a556-e8d79a41c5d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:59.196009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.195991 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb28498-fa64-4247-a556-e8d79a41c5d8-kube-api-access-8fcr8" pod "ceb28498-fa64-4247-a556-e8d79a41c5d8" (UID: "ceb28498-fa64-4247-a556-e8d79a41c5d8"). InnerVolumeSpecName "kube-api-access-8fcr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:59.216241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.216215 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc"] Oct 30 18:30:59.221626 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:30:59.219771 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32086a5d_e428_4c95_be26_8aa59580733d.slice/crio-28f2d2eb144f725dd984a3152bf2dbf8b4b9b00896dbffeebf0b55dde1122781 WatchSource:0}: Error finding container 28f2d2eb144f725dd984a3152bf2dbf8b4b9b00896dbffeebf0b55dde1122781: Status 404 returned error can't find the container with id 28f2d2eb144f725dd984a3152bf2dbf8b4b9b00896dbffeebf0b55dde1122781 Oct 30 18:30:59.295882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.295853 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb28498-fa64-4247-a556-e8d79a41c5d8-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:59.295882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.295882 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fcr8\" (UniqueName: \"kubernetes.io/projected/ceb28498-fa64-4247-a556-e8d79a41c5d8-kube-api-access-8fcr8\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:30:59.922629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.922596 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" event={"ID":"32086a5d-e428-4c95-be26-8aa59580733d","Type":"ContainerStarted","Data":"f452a49edadb6c54be429ae02aae6ac57e69938b5eabacb7192b93aaa6de71bd"} Oct 30 18:30:59.923076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.922648 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" event={"ID":"32086a5d-e428-4c95-be26-8aa59580733d","Type":"ContainerStarted","Data":"28f2d2eb144f725dd984a3152bf2dbf8b4b9b00896dbffeebf0b55dde1122781"} Oct 30 18:30:59.924052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.924027 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" event={"ID":"ceb28498-fa64-4247-a556-e8d79a41c5d8","Type":"ContainerDied","Data":"1781a999512db264239c1e1fa12f8e7a897496107fc267325e17623b39e55320"} Oct 30 18:30:59.924118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.924075 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1781a999512db264239c1e1fa12f8e7a897496107fc267325e17623b39e55320" Oct 30 18:30:59.924118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.924034 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r" Oct 30 18:30:59.959651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:30:59.959597 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" podStartSLOduration=1.959580106 podStartE2EDuration="1.959580106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:59.957811972 +0000 UTC m=+1956.782206284" watchObservedRunningTime="2026-10-30 18:30:59.959580106 +0000 UTC m=+1956.783974398" Oct 30 18:31:00.966845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:00.963356 4582 generic.go:356] "Generic (PLEG): container finished" podID="32086a5d-e428-4c95-be26-8aa59580733d" containerID="f452a49edadb6c54be429ae02aae6ac57e69938b5eabacb7192b93aaa6de71bd" exitCode=0 Oct 30 18:31:00.966845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:00.963622 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" event={"ID":"32086a5d-e428-4c95-be26-8aa59580733d","Type":"ContainerDied","Data":"f452a49edadb6c54be429ae02aae6ac57e69938b5eabacb7192b93aaa6de71bd"} Oct 30 18:31:01.072846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:01.072208 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:02.128928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.128899 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:31:02.212638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.212604 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-openshift-ca\") pod \"32086a5d-e428-4c95-be26-8aa59580733d\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " Oct 30 18:31:02.212638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.212639 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-xks-ca\") pod \"32086a5d-e428-4c95-be26-8aa59580733d\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " Oct 30 18:31:02.212894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.212683 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32086a5d-e428-4c95-be26-8aa59580733d-kube-api-access-gzxfc\" (UniqueName: \"kubernetes.io/projected/32086a5d-e428-4c95-be26-8aa59580733d-kube-api-access-gzxfc\") pod \"32086a5d-e428-4c95-be26-8aa59580733d\" (UID: \"32086a5d-e428-4c95-be26-8aa59580733d\") " Oct 30 18:31:02.213095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.213065 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-openshift-ca" pod "32086a5d-e428-4c95-be26-8aa59580733d" (UID: "32086a5d-e428-4c95-be26-8aa59580733d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:02.214848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.214790 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-xks-ca" pod "32086a5d-e428-4c95-be26-8aa59580733d" (UID: "32086a5d-e428-4c95-be26-8aa59580733d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:02.214848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.214802 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32086a5d-e428-4c95-be26-8aa59580733d-kube-api-access-gzxfc" pod "32086a5d-e428-4c95-be26-8aa59580733d" (UID: "32086a5d-e428-4c95-be26-8aa59580733d"). InnerVolumeSpecName "kube-api-access-gzxfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:02.314848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.314781 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:31:02.314848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.314816 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32086a5d-e428-4c95-be26-8aa59580733d-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:31:02.314848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.314851 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzxfc\" (UniqueName: \"kubernetes.io/projected/32086a5d-e428-4c95-be26-8aa59580733d-kube-api-access-gzxfc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:31:02.979536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.979502 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" event={"ID":"32086a5d-e428-4c95-be26-8aa59580733d","Type":"ContainerDied","Data":"28f2d2eb144f725dd984a3152bf2dbf8b4b9b00896dbffeebf0b55dde1122781"} Oct 30 18:31:02.979716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.979556 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28f2d2eb144f725dd984a3152bf2dbf8b4b9b00896dbffeebf0b55dde1122781" Oct 30 18:31:02.979716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:02.979508 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc" Oct 30 18:31:03.367440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.367344 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc"] Oct 30 18:31:03.368925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.368895 4582 cpu_manager.go:414] "containerMap: removing container" podUID="ceb28498-fa64-4247-a556-e8d79a41c5d8" containerName="revoke-keys" Oct 30 18:31:03.369075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.368955 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb28498-fa64-4247-a556-e8d79a41c5d8" containerName="revoke-keys" Oct 30 18:31:03.369075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.368973 4582 cpu_manager.go:414] "containerMap: removing container" podUID="32086a5d-e428-4c95-be26-8aa59580733d" containerName="revoke-keys" Oct 30 18:31:03.369075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.368980 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32086a5d-e428-4c95-be26-8aa59580733d" containerName="revoke-keys" Oct 30 18:31:03.369243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.369191 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb28498-fa64-4247-a556-e8d79a41c5d8" containerName="revoke-keys" Oct 30 18:31:03.369307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.369242 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="32086a5d-e428-4c95-be26-8aa59580733d" containerName="revoke-keys" Oct 30 18:31:03.372861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.372838 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.385472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.385443 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc"] Oct 30 18:31:03.390150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.390123 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ab244c-serving-cert\"" Oct 30 18:31:03.390287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.390251 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:31:03.390440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.390415 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wjb5t\"" Oct 30 18:31:03.391349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.391326 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:31:03.391498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.391444 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:31:03.437675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.437642 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:03.519777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.519746 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:03.530237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.530185 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.530407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.530391 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.530491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.530457 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.530553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.530499 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whff\" (UniqueName: \"kubernetes.io/projected/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-kube-api-access-2whff\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.632383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.632311 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.632588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.632395 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.632588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.632431 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whff\" (UniqueName: \"kubernetes.io/projected/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-kube-api-access-2whff\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.632588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.632551 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.633316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.633288 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.636309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.636271 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.638361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.638335 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.650164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.650135 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whff\" (UniqueName: \"kubernetes.io/projected/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-kube-api-access-2whff\") pod \"maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.690839 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.690796 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:03.996130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:03.996106 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc"] Oct 30 18:31:03.998273 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:03.998246 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad19fb9_c8d7_48f0_9d05_992cafb88bbc.slice/crio-c72ddd10e5acbbd99a514e7e8f03af28256ec79387c0d2d1d2ecf1f0c0ed2755 WatchSource:0}: Error finding container c72ddd10e5acbbd99a514e7e8f03af28256ec79387c0d2d1d2ecf1f0c0ed2755: Status 404 returned error can't find the container with id c72ddd10e5acbbd99a514e7e8f03af28256ec79387c0d2d1d2ecf1f0c0ed2755 Oct 30 18:31:04.995777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:04.995746 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" event={"ID":"cad19fb9-c8d7-48f0-9d05-992cafb88bbc","Type":"ContainerStarted","Data":"61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69"} Oct 30 18:31:04.996191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:04.995794 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" event={"ID":"cad19fb9-c8d7-48f0-9d05-992cafb88bbc","Type":"ContainerStarted","Data":"c72ddd10e5acbbd99a514e7e8f03af28256ec79387c0d2d1d2ecf1f0c0ed2755"} Oct 30 18:31:04.996191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:04.995949 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:05.030625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:05.030571 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" podStartSLOduration=1.545486587 podStartE2EDuration="2.030558958s" totalImagesPullingTime="485.072371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:05.029261862 +0000 UTC m=+1961.853656155" watchObservedRunningTime="2026-10-30 18:31:05.030558958 +0000 UTC m=+1961.854953249" Oct 30 18:31:05.300923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:05.300843 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:05.891381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:05.891352 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 18:31:06.227054 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:31:06.226968 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:31:07.068577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:07.068537 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:07.205590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:07.205555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:07.712335 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:07.712297 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:07.830979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:07.830936 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:08.601992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.601958 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm"] Oct 30 18:31:08.606414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.606391 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.619733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.619710 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.619871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.619768 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bll7t\" (UniqueName: \"kubernetes.io/projected/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-kube-api-access-bll7t\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.620010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.619988 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.628395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.628368 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:31:08.628557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.628424 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:31:08.635308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.635281 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm"] Oct 30 18:31:08.721349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.721305 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.721856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.721467 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.721856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.721540 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bll7t\" (UniqueName: \"kubernetes.io/projected/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-kube-api-access-bll7t\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.722050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.722030 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.723906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.723880 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.737423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.737397 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bll7t\" (UniqueName: \"kubernetes.io/projected/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-kube-api-access-bll7t\") pod \"maas-api-revoke-sub-8dec613b-766zm\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:08.921312 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:08.921282 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:09.113475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:09.113444 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:09.243149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:09.243124 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm"] Oct 30 18:31:09.245770 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:09.245739 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363c20bc_90e6_45c8_9421_fa9d1e55c3f9.slice/crio-3a5eb4d2b0f4f851baacc4c369dc2ce16d1be43932dd83829216ce3cfe2a171a WatchSource:0}: Error finding container 3a5eb4d2b0f4f851baacc4c369dc2ce16d1be43932dd83829216ce3cfe2a171a: Status 404 returned error can't find the container with id 3a5eb4d2b0f4f851baacc4c369dc2ce16d1be43932dd83829216ce3cfe2a171a Oct 30 18:31:10.040667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:10.040629 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" event={"ID":"363c20bc-90e6-45c8-9421-fa9d1e55c3f9","Type":"ContainerStarted","Data":"e238cb8d1da25471df22b5bd098ef03f3b35c4e27d4506469cebe96dbb05c83b"} Oct 30 18:31:10.041118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:10.040690 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" event={"ID":"363c20bc-90e6-45c8-9421-fa9d1e55c3f9","Type":"ContainerStarted","Data":"3a5eb4d2b0f4f851baacc4c369dc2ce16d1be43932dd83829216ce3cfe2a171a"} Oct 30 18:31:10.077257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:10.077189 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" podStartSLOduration=2.077171828 podStartE2EDuration="2.077171828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:10.074970518 +0000 UTC m=+1966.899364813" watchObservedRunningTime="2026-10-30 18:31:10.077171828 +0000 UTC m=+1966.901566121" Oct 30 18:31:11.008471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:11.008439 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 18:31:11.051537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:11.051504 4582 generic.go:356] "Generic (PLEG): container finished" podID="363c20bc-90e6-45c8-9421-fa9d1e55c3f9" containerID="e238cb8d1da25471df22b5bd098ef03f3b35c4e27d4506469cebe96dbb05c83b" exitCode=0 Oct 30 18:31:11.051951 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:11.051581 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" event={"ID":"363c20bc-90e6-45c8-9421-fa9d1e55c3f9","Type":"ContainerDied","Data":"e238cb8d1da25471df22b5bd098ef03f3b35c4e27d4506469cebe96dbb05c83b"} Oct 30 18:31:11.053283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:11.053258 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:11.374755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:11.374705 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:11.474559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:11.474524 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:11.575207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:11.575163 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:12.222006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.221981 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:12.264718 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.264682 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-xks-ca\") pod \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " Oct 30 18:31:12.264952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.264746 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-kube-api-access-bll7t\" (UniqueName: \"kubernetes.io/projected/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-kube-api-access-bll7t\") pod \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " Oct 30 18:31:12.264952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.264791 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-openshift-ca\") pod \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\" (UID: \"363c20bc-90e6-45c8-9421-fa9d1e55c3f9\") " Oct 30 18:31:12.265527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.265497 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-openshift-ca" pod "363c20bc-90e6-45c8-9421-fa9d1e55c3f9" (UID: "363c20bc-90e6-45c8-9421-fa9d1e55c3f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:12.267184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.267156 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-kube-api-access-bll7t" pod "363c20bc-90e6-45c8-9421-fa9d1e55c3f9" (UID: "363c20bc-90e6-45c8-9421-fa9d1e55c3f9"). InnerVolumeSpecName "kube-api-access-bll7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:12.267316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.267299 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-xks-ca" pod "363c20bc-90e6-45c8-9421-fa9d1e55c3f9" (UID: "363c20bc-90e6-45c8-9421-fa9d1e55c3f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:12.366751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.366655 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:31:12.366751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.366686 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bll7t\" (UniqueName: \"kubernetes.io/projected/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-kube-api-access-bll7t\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:31:12.366751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.366696 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363c20bc-90e6-45c8-9421-fa9d1e55c3f9-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:31:12.982508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:12.982441 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:13.068353 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.068318 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" event={"ID":"363c20bc-90e6-45c8-9421-fa9d1e55c3f9","Type":"ContainerDied","Data":"3a5eb4d2b0f4f851baacc4c369dc2ce16d1be43932dd83829216ce3cfe2a171a"} Oct 30 18:31:13.068528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.068362 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm" Oct 30 18:31:13.068528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.068373 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a5eb4d2b0f4f851baacc4c369dc2ce16d1be43932dd83829216ce3cfe2a171a" Oct 30 18:31:13.085548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.085498 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:13.184223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.184192 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:13.267427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.267348 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr"] Oct 30 18:31:13.268068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.268051 4582 cpu_manager.go:414] "containerMap: removing container" podUID="363c20bc-90e6-45c8-9421-fa9d1e55c3f9" containerName="revoke-keys" Oct 30 18:31:13.268131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.268069 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363c20bc-90e6-45c8-9421-fa9d1e55c3f9" containerName="revoke-keys" Oct 30 18:31:13.268190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.268181 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="363c20bc-90e6-45c8-9421-fa9d1e55c3f9" containerName="revoke-keys" Oct 30 18:31:13.271770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.271754 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.289987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.289969 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-675070\"" Oct 30 18:31:13.301888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.301863 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr"] Oct 30 18:31:13.303218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.303191 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:13.328473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.328440 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f"] Oct 30 18:31:13.334944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.334923 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.349468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.349447 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f"] Oct 30 18:31:13.352768 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.352749 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-675070-dockercfg-77969\"" Oct 30 18:31:13.379834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.379793 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c87fb8-c070-4a6d-9aad-29237f31941b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.380016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.379991 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d985j\" (UniqueName: \"kubernetes.io/projected/b2c87fb8-c070-4a6d-9aad-29237f31941b-kube-api-access-d985j\") pod \"payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.380095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.380036 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2nqf\" (UniqueName: \"kubernetes.io/projected/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-kube-api-access-m2nqf\") pod \"payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.380370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.380248 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.481707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.481673 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c87fb8-c070-4a6d-9aad-29237f31941b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.481912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.481775 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d985j\" (UniqueName: \"kubernetes.io/projected/b2c87fb8-c070-4a6d-9aad-29237f31941b-kube-api-access-d985j\") pod \"payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.481912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.481800 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2nqf\" (UniqueName: \"kubernetes.io/projected/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-kube-api-access-m2nqf\") pod \"payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.481912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.481875 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.482532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.482512 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c87fb8-c070-4a6d-9aad-29237f31941b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.482639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.482619 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.499676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.499641 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2nqf\" (UniqueName: \"kubernetes.io/projected/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-kube-api-access-m2nqf\") pod \"payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.499676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.499671 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d985j\" (UniqueName: \"kubernetes.io/projected/b2c87fb8-c070-4a6d-9aad-29237f31941b-kube-api-access-d985j\") pod \"payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.514689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.514658 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:13.599925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.599228 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:13.680604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.680572 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:13.700053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:13.699527 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:14.012172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:14.012149 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr"] Oct 30 18:31:14.013928 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:14.013902 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824ccad0_a794_4ae6_ab9c_b5cf75fda61f.slice/crio-ad4a26913d3d5eb52b3f11eaedf7587b9874de3e7b50bab885867bcdca1e9161 WatchSource:0}: Error finding container ad4a26913d3d5eb52b3f11eaedf7587b9874de3e7b50bab885867bcdca1e9161: Status 404 returned error can't find the container with id ad4a26913d3d5eb52b3f11eaedf7587b9874de3e7b50bab885867bcdca1e9161 Oct 30 18:31:14.033759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:14.033695 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f"] Oct 30 18:31:14.035709 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:14.035683 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c87fb8_c070_4a6d_9aad_29237f31941b.slice/crio-fd28c517bb5866ddd6b7cf17b25ea59f0f7f5ae3081e0eb8114e49f441749673 WatchSource:0}: Error finding container fd28c517bb5866ddd6b7cf17b25ea59f0f7f5ae3081e0eb8114e49f441749673: Status 404 returned error can't find the container with id fd28c517bb5866ddd6b7cf17b25ea59f0f7f5ae3081e0eb8114e49f441749673 Oct 30 18:31:14.079880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:14.079848 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" event={"ID":"824ccad0-a794-4ae6-ab9c-b5cf75fda61f","Type":"ContainerStarted","Data":"ad4a26913d3d5eb52b3f11eaedf7587b9874de3e7b50bab885867bcdca1e9161"} Oct 30 18:31:14.087007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:14.086967 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" event={"ID":"b2c87fb8-c070-4a6d-9aad-29237f31941b","Type":"ContainerStarted","Data":"fd28c517bb5866ddd6b7cf17b25ea59f0f7f5ae3081e0eb8114e49f441749673"} Oct 30 18:31:15.034056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.034021 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:15.096753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.096716 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" event={"ID":"b2c87fb8-c070-4a6d-9aad-29237f31941b","Type":"ContainerStarted","Data":"3a05d0575f10252c7d6c266f592b17a45bb462afeea4c94bd387b3f03aa4ecb0"} Oct 30 18:31:15.096974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.096807 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:15.098539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.098514 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" event={"ID":"824ccad0-a794-4ae6-ab9c-b5cf75fda61f","Type":"ContainerStarted","Data":"b691225f9ba9e06dfa530b6507f6d20de4362a9f8164fea3365ffcc442a6988c"} Oct 30 18:31:15.098641 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.098596 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:15.134467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.134415 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" podStartSLOduration=2.13440145 podStartE2EDuration="2.13440145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:15.131931007 +0000 UTC m=+1971.956325299" watchObservedRunningTime="2026-10-30 18:31:15.13440145 +0000 UTC m=+1971.958795741" Oct 30 18:31:15.136197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.136169 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:15.164714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.164666 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" podStartSLOduration=2.164657634 podStartE2EDuration="2.164657634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:15.16368235 +0000 UTC m=+1971.988076642" watchObservedRunningTime="2026-10-30 18:31:15.164657634 +0000 UTC m=+1971.989051925" Oct 30 18:31:15.236793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:15.236763 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:16.807996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:16.807968 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:17.527590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:17.527557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:17.628848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:17.628786 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:18.928049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:18.928011 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:19.618843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:19.618318 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:20.072776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.072589 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:20.246849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.243643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:20.579845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.577790 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:20.692320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.692222 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:20.852685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.852586 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p"] Oct 30 18:31:20.960844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.959584 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:20.982845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.972435 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p"] Oct 30 18:31:20.999842 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:20.999791 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-1a2436-dockercfg-dmtgd\"" Oct 30 18:31:21.052892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.052814 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-1a2436\"" Oct 30 18:31:21.066602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.066564 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bwf\" (UniqueName: \"kubernetes.io/projected/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-kube-api-access-82bwf\") pod \"payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:21.068289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.068264 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:21.169873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.169437 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bwf\" (UniqueName: \"kubernetes.io/projected/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-kube-api-access-82bwf\") pod \"payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:21.169873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.169481 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:21.170303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.170271 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:21.186249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.186216 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bwf\" (UniqueName: \"kubernetes.io/projected/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-kube-api-access-82bwf\") pod \"payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:21.379878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.379846 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:21.677415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:21.677381 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p"] Oct 30 18:31:21.679249 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:21.679223 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5936235_bcfc_4c16_949c_b7e5a1b6fb35.slice/crio-6c6493b006f77277bc4ccff7fcbf13b7bc690a0591e9f82432032aa48179dcfc WatchSource:0}: Error finding container 6c6493b006f77277bc4ccff7fcbf13b7bc690a0591e9f82432032aa48179dcfc: Status 404 returned error can't find the container with id 6c6493b006f77277bc4ccff7fcbf13b7bc690a0591e9f82432032aa48179dcfc Oct 30 18:31:22.203838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:22.203796 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" event={"ID":"e5936235-bcfc-4c16-949c-b7e5a1b6fb35","Type":"ContainerStarted","Data":"d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9"} Oct 30 18:31:22.204337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:22.203873 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" event={"ID":"e5936235-bcfc-4c16-949c-b7e5a1b6fb35","Type":"ContainerStarted","Data":"6c6493b006f77277bc4ccff7fcbf13b7bc690a0591e9f82432032aa48179dcfc"} Oct 30 18:31:22.204337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:22.203938 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:22.236955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:22.236903 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" podStartSLOduration=2.236890693 podStartE2EDuration="2.236890693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:22.236769186 +0000 UTC m=+1979.061163477" watchObservedRunningTime="2026-10-30 18:31:22.236890693 +0000 UTC m=+1979.061284982" Oct 30 18:31:22.638537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:22.638496 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:23.030076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:23.029995 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg"] Oct 30 18:31:23.038431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:23.038407 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847168d2-xbmlg"] Oct 30 18:31:23.303100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:23.303022 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd" path="/var/lib/kubelet/pods/45f3e58a-33af-4e84-8dea-7a4c9f9ae2bd/volumes" Oct 30 18:31:24.875435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:24.875386 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:26.109673 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.109644 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 18:31:26.110105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.109993 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 18:31:26.802252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.802221 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl"] Oct 30 18:31:26.807868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.807847 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:26.829747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.829718 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-675070-serving-cert\"" Oct 30 18:31:26.834274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.834241 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl"] Oct 30 18:31:26.895996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.895966 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:26.942762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.942732 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:26.942946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.942800 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:26.942946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.942928 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a928e3-74fc-442e-a69d-794dceb6774c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:26.943087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.943070 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8h4\" (UniqueName: \"kubernetes.io/projected/21a928e3-74fc-442e-a69d-794dceb6774c-kube-api-access-tx8h4\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:26.995968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:26.995940 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:27.044218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.044187 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.044369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.044235 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a928e3-74fc-442e-a69d-794dceb6774c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.044423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.044365 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8h4\" (UniqueName: \"kubernetes.io/projected/21a928e3-74fc-442e-a69d-794dceb6774c-kube-api-access-tx8h4\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.044529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.044511 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.045084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.045061 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a928e3-74fc-442e-a69d-794dceb6774c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.046699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.046675 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.046850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.046814 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.061675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.061627 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8h4\" (UniqueName: \"kubernetes.io/projected/21a928e3-74fc-442e-a69d-794dceb6774c-kube-api-access-tx8h4\") pod \"maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.098499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.098462 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:27.122908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.122220 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:27.426324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.426294 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl"] Oct 30 18:31:27.432404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:27.432382 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:31:28.263080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:28.262545 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" event={"ID":"21a928e3-74fc-442e-a69d-794dceb6774c","Type":"ContainerStarted","Data":"bc81e7f2f8bfec4c968cc9a344a13fd851d609b4a67dcb2815eb65f513f25871"} Oct 30 18:31:28.263080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:28.263023 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:28.263663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:28.263636 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" event={"ID":"21a928e3-74fc-442e-a69d-794dceb6774c","Type":"ContainerStarted","Data":"2bfc57586a5197622b8ea696cf93074484467db6a95fddc3c3699e3e29927335"} Oct 30 18:31:28.299134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:28.299084 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" podStartSLOduration=1.996026521 podStartE2EDuration="2.299069935s" totalImagesPullingTime="303.043414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:28.297443051 +0000 UTC m=+1985.121837342" watchObservedRunningTime="2026-10-30 18:31:28.299069935 +0000 UTC m=+1985.123464227" Oct 30 18:31:29.235681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:29.235645 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:30.308485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:30.308457 4582 scope.go:122] "RemoveContainer" containerID="e18ccac4106262fff0f538c791267875ce53a258d90767b90104c346edbcbcc3" Oct 30 18:31:30.318647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:30.318629 4582 scope.go:122] "RemoveContainer" containerID="5a871d6ed599e42c11cb2211bd84e2c12c961e2324e1f2f7945dcd8e425a3131" Oct 30 18:31:30.327601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:30.327583 4582 scope.go:122] "RemoveContainer" containerID="ef26a3fb3f7e8a8491e16fcc998600c112343472016f97604a5af40c7ab75b75" Oct 30 18:31:31.182915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:31.182881 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:31.358399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:31.358367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:31.847615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:31.847577 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:32.032989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:32.032954 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:32.245487 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:32.245452 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:32.700813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:32.700775 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:33.199511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:33.199474 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:33.216248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:33.216222 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 18:31:33.401388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:33.401354 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:33.498650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:33.498564 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:34.275460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:34.275435 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 18:31:35.327456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.327424 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z"] Oct 30 18:31:35.331627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.331603 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.350084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.350064 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-1a2436-serving-cert\"" Oct 30 18:31:35.367298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.367269 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z"] Oct 30 18:31:35.410082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.410049 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:35.415078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.415044 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.415243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.415223 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgbt\" (UniqueName: \"kubernetes.io/projected/94045cce-08d4-4f95-8af7-5dd271d8ef96-kube-api-access-ckgbt\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.415388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.415373 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.415434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.415414 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94045cce-08d4-4f95-8af7-5dd271d8ef96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.490337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.490304 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:35.516266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.516241 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.516442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.516307 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94045cce-08d4-4f95-8af7-5dd271d8ef96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.516442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.516370 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.516566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.516512 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgbt\" (UniqueName: \"kubernetes.io/projected/94045cce-08d4-4f95-8af7-5dd271d8ef96-kube-api-access-ckgbt\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.517010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.516984 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94045cce-08d4-4f95-8af7-5dd271d8ef96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.518765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.518743 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.518883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.518772 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.532913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.532883 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgbt\" (UniqueName: \"kubernetes.io/projected/94045cce-08d4-4f95-8af7-5dd271d8ef96-kube-api-access-ckgbt\") pod \"maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.590587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.590516 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:35.646521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.646491 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:35.889852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.889807 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:35.956432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:35.956394 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z"] Oct 30 18:31:35.958453 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:35.958428 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94045cce_08d4_4f95_8af7_5dd271d8ef96.slice/crio-c5059543d8aa819ce9fd840bd68a8f728399a798c89af1ee641029803e55d0a7 WatchSource:0}: Error finding container c5059543d8aa819ce9fd840bd68a8f728399a798c89af1ee641029803e55d0a7: Status 404 returned error can't find the container with id c5059543d8aa819ce9fd840bd68a8f728399a798c89af1ee641029803e55d0a7 Oct 30 18:31:36.022131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:36.022099 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:36.137292 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:36.137257 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:36.226274 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:31:36.226182 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:31:36.422773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:36.422730 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" event={"ID":"94045cce-08d4-4f95-8af7-5dd271d8ef96","Type":"ContainerStarted","Data":"5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d"} Oct 30 18:31:36.423314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:36.422802 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:36.423314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:36.422842 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" event={"ID":"94045cce-08d4-4f95-8af7-5dd271d8ef96","Type":"ContainerStarted","Data":"c5059543d8aa819ce9fd840bd68a8f728399a798c89af1ee641029803e55d0a7"} Oct 30 18:31:36.463306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:36.463253 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" podStartSLOduration=1.117975252 podStartE2EDuration="1.463236903s" totalImagesPullingTime="345.261651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:36.459133718 +0000 UTC m=+1993.283528010" watchObservedRunningTime="2026-10-30 18:31:36.463236903 +0000 UTC m=+1993.287631195" Oct 30 18:31:37.679107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.679053 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:37.777926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.777888 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:37.806618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.806588 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d"] Oct 30 18:31:37.811266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.811246 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:37.842848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.842729 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d"] Oct 30 18:31:37.844793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.844764 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-e95c51\"" Oct 30 18:31:37.875112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.875084 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh"] Oct 30 18:31:37.879433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.879414 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:37.894545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.894515 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh"] Oct 30 18:31:37.931555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.931492 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:37.956205 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.956177 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1890c4b-70b6-4783-b73f-fc6d58634d1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:37.956344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.956306 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:37.956391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.956372 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxp5\" (UniqueName: \"kubernetes.io/projected/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-kube-api-access-vqxp5\") pod \"payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:37.956444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.956430 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8m5t\" (UniqueName: \"kubernetes.io/projected/b1890c4b-70b6-4783-b73f-fc6d58634d1a-kube-api-access-b8m5t\") pod \"payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:37.974428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:37.974407 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-e95c51-dockercfg-f88w6\"" Oct 30 18:31:38.034845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.034317 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:38.059193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.059152 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8m5t\" (UniqueName: \"kubernetes.io/projected/b1890c4b-70b6-4783-b73f-fc6d58634d1a-kube-api-access-b8m5t\") pod \"payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:38.059418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.059334 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1890c4b-70b6-4783-b73f-fc6d58634d1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:38.059492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.059457 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:38.059553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.059516 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxp5\" (UniqueName: \"kubernetes.io/projected/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-kube-api-access-vqxp5\") pod \"payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:38.061437 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.060335 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:38.061437 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.061020 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1890c4b-70b6-4783-b73f-fc6d58634d1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:38.080194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.078466 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8m5t\" (UniqueName: \"kubernetes.io/projected/b1890c4b-70b6-4783-b73f-fc6d58634d1a-kube-api-access-b8m5t\") pod \"payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:38.080194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.079948 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxp5\" (UniqueName: \"kubernetes.io/projected/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-kube-api-access-vqxp5\") pod \"payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:38.129442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.129413 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:38.143546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.143516 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:38.196683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.196429 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:38.439949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.439921 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d"] Oct 30 18:31:38.442124 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:38.442088 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1890c4b_70b6_4783_b73f_fc6d58634d1a.slice/crio-22b7d4e7cf2b253586517a5aa76b67c78422231a5fcd52e13dfffb8ffc276729 WatchSource:0}: Error finding container 22b7d4e7cf2b253586517a5aa76b67c78422231a5fcd52e13dfffb8ffc276729: Status 404 returned error can't find the container with id 22b7d4e7cf2b253586517a5aa76b67c78422231a5fcd52e13dfffb8ffc276729 Oct 30 18:31:38.514300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:38.514276 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh"] Oct 30 18:31:38.517151 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:38.517123 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0845c6d_ab7d_4077_a5ba_a6d151b3272a.slice/crio-b8e71b9e9b1bddca3af7841948134c5b8009a7ad6792fd2d47cb86c1ced4c723 WatchSource:0}: Error finding container b8e71b9e9b1bddca3af7841948134c5b8009a7ad6792fd2d47cb86c1ced4c723: Status 404 returned error can't find the container with id b8e71b9e9b1bddca3af7841948134c5b8009a7ad6792fd2d47cb86c1ced4c723 Oct 30 18:31:39.449928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.449891 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" event={"ID":"c0845c6d-ab7d-4077-a5ba-a6d151b3272a","Type":"ContainerStarted","Data":"30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003"} Oct 30 18:31:39.450425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.449959 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" event={"ID":"c0845c6d-ab7d-4077-a5ba-a6d151b3272a","Type":"ContainerStarted","Data":"b8e71b9e9b1bddca3af7841948134c5b8009a7ad6792fd2d47cb86c1ced4c723"} Oct 30 18:31:39.450425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.449992 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:39.451904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.451881 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" event={"ID":"b1890c4b-70b6-4783-b73f-fc6d58634d1a","Type":"ContainerStarted","Data":"0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7"} Oct 30 18:31:39.452035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.451919 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" event={"ID":"b1890c4b-70b6-4783-b73f-fc6d58634d1a","Type":"ContainerStarted","Data":"22b7d4e7cf2b253586517a5aa76b67c78422231a5fcd52e13dfffb8ffc276729"} Oct 30 18:31:39.452035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.452019 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:39.487923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.487875 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" podStartSLOduration=2.487860224 podStartE2EDuration="2.487860224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:39.483495604 +0000 UTC m=+1996.307889909" watchObservedRunningTime="2026-10-30 18:31:39.487860224 +0000 UTC m=+1996.312254517" Oct 30 18:31:39.518496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:39.518444 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" podStartSLOduration=2.518433415 podStartE2EDuration="2.518433415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:39.517985142 +0000 UTC m=+1996.342379434" watchObservedRunningTime="2026-10-30 18:31:39.518433415 +0000 UTC m=+1996.342827707" Oct 30 18:31:40.107096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.107062 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:40.206184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.206147 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:40.232420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.232387 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq"] Oct 30 18:31:40.237082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.237062 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.254608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.254586 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq"] Oct 30 18:31:40.258934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.258916 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4a7508\"" Oct 30 18:31:40.295769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.295738 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4"] Oct 30 18:31:40.301154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.301136 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.314505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.314484 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4a7508-dockercfg-q8t4h\"" Oct 30 18:31:40.317874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.317848 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4"] Oct 30 18:31:40.422845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.412169 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f410e59-f1d4-4741-9d50-ba48d7b67588-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.422845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.412228 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ttw\" (UniqueName: \"kubernetes.io/projected/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-kube-api-access-t7ttw\") pod \"payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.422845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.412310 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.422845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.412353 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjdl\" (UniqueName: \"kubernetes.io/projected/1f410e59-f1d4-4741-9d50-ba48d7b67588-kube-api-access-zdjdl\") pod \"payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.513397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.513358 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f410e59-f1d4-4741-9d50-ba48d7b67588-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.513934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.513424 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ttw\" (UniqueName: \"kubernetes.io/projected/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-kube-api-access-t7ttw\") pod \"payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.513934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.513504 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.513934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.513647 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjdl\" (UniqueName: \"kubernetes.io/projected/1f410e59-f1d4-4741-9d50-ba48d7b67588-kube-api-access-zdjdl\") pod \"payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.514291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.514262 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f410e59-f1d4-4741-9d50-ba48d7b67588-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.514368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.514311 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.525410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.525386 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:40.530367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.530344 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjdl\" (UniqueName: \"kubernetes.io/projected/1f410e59-f1d4-4741-9d50-ba48d7b67588-kube-api-access-zdjdl\") pod \"payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.530367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.530367 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ttw\" (UniqueName: \"kubernetes.io/projected/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-kube-api-access-t7ttw\") pod \"payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.551593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.551569 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:40.614984 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.614951 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:40.857889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.857864 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq"] Oct 30 18:31:40.859902 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:40.859874 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f410e59_f1d4_4741_9d50_ba48d7b67588.slice/crio-dc3fedf8b1904cb477b2449c853a5133090d68b179a0160b0c06fb5243aa5dc5 WatchSource:0}: Error finding container dc3fedf8b1904cb477b2449c853a5133090d68b179a0160b0c06fb5243aa5dc5: Status 404 returned error can't find the container with id dc3fedf8b1904cb477b2449c853a5133090d68b179a0160b0c06fb5243aa5dc5 Oct 30 18:31:40.930554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:40.930531 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4"] Oct 30 18:31:40.931972 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:40.931942 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7b9c7a_5c24_4a9b_aba9_0bc677a73a3e.slice/crio-805d075b433e3da9ea70c4715fc2b7802a85630c0bb80c12a5c396edc0b7fce8 WatchSource:0}: Error finding container 805d075b433e3da9ea70c4715fc2b7802a85630c0bb80c12a5c396edc0b7fce8: Status 404 returned error can't find the container with id 805d075b433e3da9ea70c4715fc2b7802a85630c0bb80c12a5c396edc0b7fce8 Oct 30 18:31:41.475902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.475865 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" event={"ID":"1f410e59-f1d4-4741-9d50-ba48d7b67588","Type":"ContainerStarted","Data":"9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec"} Oct 30 18:31:41.476096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.475927 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" event={"ID":"1f410e59-f1d4-4741-9d50-ba48d7b67588","Type":"ContainerStarted","Data":"dc3fedf8b1904cb477b2449c853a5133090d68b179a0160b0c06fb5243aa5dc5"} Oct 30 18:31:41.476096 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.475962 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:41.477435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.477408 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" event={"ID":"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e","Type":"ContainerStarted","Data":"7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5"} Oct 30 18:31:41.477552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.477450 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" event={"ID":"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e","Type":"ContainerStarted","Data":"805d075b433e3da9ea70c4715fc2b7802a85630c0bb80c12a5c396edc0b7fce8"} Oct 30 18:31:41.477552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.477476 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:41.515466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.515416 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" podStartSLOduration=1.515400895 podStartE2EDuration="1.515400895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:41.511452424 +0000 UTC m=+1998.335846715" watchObservedRunningTime="2026-10-30 18:31:41.515400895 +0000 UTC m=+1998.339795187" Oct 30 18:31:41.547898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:41.547027 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" podStartSLOduration=1.547013416 podStartE2EDuration="1.547013416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:41.544086638 +0000 UTC m=+1998.368480930" watchObservedRunningTime="2026-10-30 18:31:41.547013416 +0000 UTC m=+1998.371407708" Oct 30 18:31:42.437685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:42.437651 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 18:31:42.918900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:42.918867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:44.738465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:44.738435 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:45.892785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:45.892751 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:45.994152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:45.994118 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:46.145004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:46.144919 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:46.258725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:46.258694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:46.987803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:46.987766 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:48.001216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.001178 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:48.107614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.107583 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:48.191539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.191499 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn"] Oct 30 18:31:48.197705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.197679 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.215392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.215363 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn"] Oct 30 18:31:48.217350 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.217326 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1e0b31\"" Oct 30 18:31:48.245570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.245529 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1e0b31-dockercfg-nlv8j\"" Oct 30 18:31:48.339528 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.339438 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4hf\" (UniqueName: \"kubernetes.io/projected/0d808add-11cf-44dc-9476-f1c8f7138f63-kube-api-access-7c4hf\") pod \"payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.340440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.340412 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d808add-11cf-44dc-9476-f1c8f7138f63-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.360904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.360805 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp"] Oct 30 18:31:48.367208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.367074 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.368632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.368481 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp"] Oct 30 18:31:48.442399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.442339 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4hf\" (UniqueName: \"kubernetes.io/projected/0d808add-11cf-44dc-9476-f1c8f7138f63-kube-api-access-7c4hf\") pod \"payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.442852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.442808 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d808add-11cf-44dc-9476-f1c8f7138f63-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.442980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.442952 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0648deb-e81d-4a66-b358-574a744136b2-plugins-config-volume\") pod \"payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.443062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.443045 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxxv\" (UniqueName: \"kubernetes.io/projected/e0648deb-e81d-4a66-b358-574a744136b2-kube-api-access-wnxxv\") pod \"payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.443912 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.443881 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d808add-11cf-44dc-9476-f1c8f7138f63-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.463042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.463018 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4hf\" (UniqueName: \"kubernetes.io/projected/0d808add-11cf-44dc-9476-f1c8f7138f63-kube-api-access-7c4hf\") pod \"payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.518476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.518442 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:48.544452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.544384 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0648deb-e81d-4a66-b358-574a744136b2-plugins-config-volume\") pod \"payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.544583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.544484 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxxv\" (UniqueName: \"kubernetes.io/projected/e0648deb-e81d-4a66-b358-574a744136b2-kube-api-access-wnxxv\") pod \"payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.545452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.545309 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0648deb-e81d-4a66-b358-574a744136b2-plugins-config-volume\") pod \"payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.562376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.562342 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxxv\" (UniqueName: \"kubernetes.io/projected/e0648deb-e81d-4a66-b358-574a744136b2-kube-api-access-wnxxv\") pod \"payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.687870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.687837 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:48.841158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.841091 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn"] Oct 30 18:31:48.842616 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:48.842588 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d808add_11cf_44dc_9476_f1c8f7138f63.slice/crio-5bf5334a265d4599a58e41aaa6d2b680f5f5b8f6f3b069139f86ab7df8ccfdb6 WatchSource:0}: Error finding container 5bf5334a265d4599a58e41aaa6d2b680f5f5b8f6f3b069139f86ab7df8ccfdb6: Status 404 returned error can't find the container with id 5bf5334a265d4599a58e41aaa6d2b680f5f5b8f6f3b069139f86ab7df8ccfdb6 Oct 30 18:31:48.994677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:48.994651 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp"] Oct 30 18:31:48.997435 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:48.997404 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0648deb_e81d_4a66_b358_574a744136b2.slice/crio-d7809da07fa0156753c9f6faec97814f004ad87abe078bfe751a5d702f4cdd26 WatchSource:0}: Error finding container d7809da07fa0156753c9f6faec97814f004ad87abe078bfe751a5d702f4cdd26: Status 404 returned error can't find the container with id d7809da07fa0156753c9f6faec97814f004ad87abe078bfe751a5d702f4cdd26 Oct 30 18:31:49.544140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.544101 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" event={"ID":"0d808add-11cf-44dc-9476-f1c8f7138f63","Type":"ContainerStarted","Data":"acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e"} Oct 30 18:31:49.544515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.544164 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:31:49.544515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.544187 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" event={"ID":"0d808add-11cf-44dc-9476-f1c8f7138f63","Type":"ContainerStarted","Data":"5bf5334a265d4599a58e41aaa6d2b680f5f5b8f6f3b069139f86ab7df8ccfdb6"} Oct 30 18:31:49.545775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.545749 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" event={"ID":"e0648deb-e81d-4a66-b358-574a744136b2","Type":"ContainerStarted","Data":"6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2"} Oct 30 18:31:49.545928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.545796 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:31:49.545928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.545812 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" event={"ID":"e0648deb-e81d-4a66-b358-574a744136b2","Type":"ContainerStarted","Data":"d7809da07fa0156753c9f6faec97814f004ad87abe078bfe751a5d702f4cdd26"} Oct 30 18:31:49.580675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.580625 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" podStartSLOduration=1.580609575 podStartE2EDuration="1.580609575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:49.57780213 +0000 UTC m=+2006.402196421" watchObservedRunningTime="2026-10-30 18:31:49.580609575 +0000 UTC m=+2006.405003867" Oct 30 18:31:49.611083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:49.611038 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" podStartSLOduration=1.6110251309999999 podStartE2EDuration="1.611025131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:49.610587309 +0000 UTC m=+2006.434981601" watchObservedRunningTime="2026-10-30 18:31:49.611025131 +0000 UTC m=+2006.435419422" Oct 30 18:31:50.200146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:50.200110 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:50.470423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:50.470342 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 18:31:50.470423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:50.470421 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 18:31:51.936558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:51.936518 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:52.380518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:52.380489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:52.488717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:52.488688 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 18:31:52.488926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:52.488802 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 18:31:52.573334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:52.573306 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:52.970515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:52.970478 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:53.368264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:53.368233 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:54.805013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:54.804981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:56.476402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.476368 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct"] Oct 30 18:31:56.481968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.481947 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.499714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.499696 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-e95c51-serving-cert\"" Oct 30 18:31:56.514979 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.514956 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct"] Oct 30 18:31:56.553748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.553719 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:56.640983 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.640953 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:56.657986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.657955 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hcl\" (UniqueName: \"kubernetes.io/projected/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-kube-api-access-h9hcl\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.658128 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.658115 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.658239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.658218 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.658298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.658261 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.744036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.743943 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:56.759479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.759445 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.759689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.759499 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.759689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.759571 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hcl\" (UniqueName: \"kubernetes.io/projected/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-kube-api-access-h9hcl\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.759861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.759728 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.762631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.762599 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.762766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.762706 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.763043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.763027 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.776926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.776901 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hcl\" (UniqueName: \"kubernetes.io/projected/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-kube-api-access-h9hcl\") pod \"maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:56.801101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:56.801077 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:57.118565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:57.118534 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct"] Oct 30 18:31:57.118873 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:31:57.118845 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbf9d38_6d75_43d5_8d16_74b3cafb40dd.slice/crio-5394c420fd7998a195f23fda0749d290ccca62f393364ab39894718f985f7e45 WatchSource:0}: Error finding container 5394c420fd7998a195f23fda0749d290ccca62f393364ab39894718f985f7e45: Status 404 returned error can't find the container with id 5394c420fd7998a195f23fda0749d290ccca62f393364ab39894718f985f7e45 Oct 30 18:31:57.613360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:57.613327 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" event={"ID":"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd","Type":"ContainerStarted","Data":"d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3"} Oct 30 18:31:57.613676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:57.613391 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" event={"ID":"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd","Type":"ContainerStarted","Data":"5394c420fd7998a195f23fda0749d290ccca62f393364ab39894718f985f7e45"} Oct 30 18:31:57.613676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:57.613419 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:31:57.657653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:57.657606 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" podStartSLOduration=1.2520678570000001 podStartE2EDuration="1.657593684s" totalImagesPullingTime="405.525827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:57.65626816 +0000 UTC m=+2014.480662463" watchObservedRunningTime="2026-10-30 18:31:57.657593684 +0000 UTC m=+2014.481987975" Oct 30 18:31:58.554247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:31:58.554216 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:00.352810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.352777 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv"] Oct 30 18:32:00.357556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.357539 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.381418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.381394 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4a7508-serving-cert\"" Oct 30 18:32:00.386775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.386752 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv"] Oct 30 18:32:00.453727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.453686 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:00.519704 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.519671 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.519900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.519813 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.519984 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.519967 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9997e625-9d15-4297-8813-cbde3f862c8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.520031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.520002 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cd7r\" (UniqueName: \"kubernetes.io/projected/9997e625-9d15-4297-8813-cbde3f862c8b-kube-api-access-4cd7r\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.558784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.558751 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:00.560041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.560021 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:32:00.560181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.560090 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:32:00.620771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.620700 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.620972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.620793 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.620972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.620937 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9997e625-9d15-4297-8813-cbde3f862c8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.621105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.620972 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cd7r\" (UniqueName: \"kubernetes.io/projected/9997e625-9d15-4297-8813-cbde3f862c8b-kube-api-access-4cd7r\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.622044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.622019 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9997e625-9d15-4297-8813-cbde3f862c8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.623343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.623320 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.624898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.624873 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.641203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.641178 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cd7r\" (UniqueName: \"kubernetes.io/projected/9997e625-9d15-4297-8813-cbde3f862c8b-kube-api-access-4cd7r\") pod \"maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.670133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.670104 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:00.673262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.673232 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:00.988064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:00.988038 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv"] Oct 30 18:32:00.990469 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:32:00.990440 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9997e625_9d15_4297_8813_cbde3f862c8b.slice/crio-1e6dce0a828905216762776a38ba362f44b8189dc7b7174fe491d8db61660793 WatchSource:0}: Error finding container 1e6dce0a828905216762776a38ba362f44b8189dc7b7174fe491d8db61660793: Status 404 returned error can't find the container with id 1e6dce0a828905216762776a38ba362f44b8189dc7b7174fe491d8db61660793 Oct 30 18:32:01.662116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:01.662034 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" event={"ID":"9997e625-9d15-4297-8813-cbde3f862c8b","Type":"ContainerStarted","Data":"1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f"} Oct 30 18:32:01.662116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:01.662086 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" event={"ID":"9997e625-9d15-4297-8813-cbde3f862c8b","Type":"ContainerStarted","Data":"1e6dce0a828905216762776a38ba362f44b8189dc7b7174fe491d8db61660793"} Oct 30 18:32:01.662631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:01.662147 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:01.699104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:01.699059 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" podStartSLOduration=1.282538752 podStartE2EDuration="1.699046038s" totalImagesPullingTime="416.507286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:01.695720842 +0000 UTC m=+2018.520115134" watchObservedRunningTime="2026-10-30 18:32:01.699046038 +0000 UTC m=+2018.523440375" Oct 30 18:32:03.265604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:03.265565 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:03.360931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:03.360894 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:03.628381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:03.628352 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 18:32:03.721447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:03.721404 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:04.190968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:04.190933 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:04.363578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:04.363545 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:05.668369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:05.668335 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:05.982391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:05.982298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:06.084735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:06.084697 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:06.225988 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:32:06.225951 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:32:06.910952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:06.910915 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:07.051452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:07.051420 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:07.154183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:07.154142 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:07.676456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:07.676430 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 18:32:08.056486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:08.056395 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:08.678028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:08.677994 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:08.789199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:08.789148 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:08.894657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:08.894627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:10.327714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.327673 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:10.571190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.571162 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:10.673175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.673138 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:10.929002 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.928911 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c"] Oct 30 18:32:10.947439 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.947391 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c"] Oct 30 18:32:10.947644 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.947499 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:10.959851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.959815 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-24b124\"" Oct 30 18:32:10.969617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:10.969589 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-24b124-dockercfg-52rlc\"" Oct 30 18:32:11.007197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.007150 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff8650-d22d-4c6f-a351-fcb70e40980f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.007391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.007221 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfsn\" (UniqueName: \"kubernetes.io/projected/66ff8650-d22d-4c6f-a351-fcb70e40980f-kube-api-access-4gfsn\") pod \"payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.108793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.108750 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff8650-d22d-4c6f-a351-fcb70e40980f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.109010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.108809 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfsn\" (UniqueName: \"kubernetes.io/projected/66ff8650-d22d-4c6f-a351-fcb70e40980f-kube-api-access-4gfsn\") pod \"payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.109473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.109453 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff8650-d22d-4c6f-a351-fcb70e40980f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.125583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.125554 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfsn\" (UniqueName: \"kubernetes.io/projected/66ff8650-d22d-4c6f-a351-fcb70e40980f-kube-api-access-4gfsn\") pod \"payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.264337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.264260 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.566057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.566029 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c"] Oct 30 18:32:11.567417 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:32:11.567393 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ff8650_d22d_4c6f_a351_fcb70e40980f.slice/crio-ab5ddddd25445c3fa3dcce8c24bed80b97a97c84f5946e0264f327c154a4dd7e WatchSource:0}: Error finding container ab5ddddd25445c3fa3dcce8c24bed80b97a97c84f5946e0264f327c154a4dd7e: Status 404 returned error can't find the container with id ab5ddddd25445c3fa3dcce8c24bed80b97a97c84f5946e0264f327c154a4dd7e Oct 30 18:32:11.752108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.752065 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" event={"ID":"66ff8650-d22d-4c6f-a351-fcb70e40980f","Type":"ContainerStarted","Data":"f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff"} Oct 30 18:32:11.752546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.752131 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" event={"ID":"66ff8650-d22d-4c6f-a351-fcb70e40980f","Type":"ContainerStarted","Data":"ab5ddddd25445c3fa3dcce8c24bed80b97a97c84f5946e0264f327c154a4dd7e"} Oct 30 18:32:11.752546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.752158 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:11.789683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:11.789635 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" podStartSLOduration=1.789621424 podStartE2EDuration="1.789621424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:11.785681927 +0000 UTC m=+2028.610076219" watchObservedRunningTime="2026-10-30 18:32:11.789621424 +0000 UTC m=+2028.614015715" Oct 30 18:32:12.253286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.253252 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f"] Oct 30 18:32:12.258008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.257989 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.276886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.276859 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f"] Oct 30 18:32:12.280660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.280637 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-1e0b31-serving-cert\"" Oct 30 18:32:12.325017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.324984 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/6684ee56-e803-430a-8049-560bac7cf26a-kube-api-access-bmgmb\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.325189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.325026 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:12.325189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.325044 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee56-e803-430a-8049-560bac7cf26a-openshift-service-ca\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.325189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.325159 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-maas-api-tls\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.325336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.325222 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-metrics-certs\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.411050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.411014 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:12.426515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.426479 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-maas-api-tls\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.426675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.426560 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-metrics-certs\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.426810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.426722 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/6684ee56-e803-430a-8049-560bac7cf26a-kube-api-access-bmgmb\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.426810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.426780 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee56-e803-430a-8049-560bac7cf26a-openshift-service-ca\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.427406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.427383 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee56-e803-430a-8049-560bac7cf26a-openshift-service-ca\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.429004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.428984 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-maas-api-tls\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.429098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.429025 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-metrics-certs\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.444525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.444495 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/6684ee56-e803-430a-8049-560bac7cf26a-kube-api-access-bmgmb\") pod \"maas-api-e2e-autores-1e0b31-8bc774c57-lp58f\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.513197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.513106 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:12.572092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.572059 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:12.882161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:12.882136 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f"] Oct 30 18:32:12.884287 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:32:12.884258 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6684ee56_e803_430a_8049_560bac7cf26a.slice/crio-9d05b0041345c7925af887d79408018566de5f521864949b642f7de61e716499 WatchSource:0}: Error finding container 9d05b0041345c7925af887d79408018566de5f521864949b642f7de61e716499: Status 404 returned error can't find the container with id 9d05b0041345c7925af887d79408018566de5f521864949b642f7de61e716499 Oct 30 18:32:13.770112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:13.770070 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" event={"ID":"6684ee56-e803-430a-8049-560bac7cf26a","Type":"ContainerStarted","Data":"e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945"} Oct 30 18:32:13.770539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:13.770140 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" event={"ID":"6684ee56-e803-430a-8049-560bac7cf26a","Type":"ContainerStarted","Data":"9d05b0041345c7925af887d79408018566de5f521864949b642f7de61e716499"} Oct 30 18:32:13.770539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:13.770170 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:13.806782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:13.806727 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" podStartSLOduration=1.328540629 podStartE2EDuration="1.806710999s" totalImagesPullingTime="478.17037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:13.805601848 +0000 UTC m=+2030.629996139" watchObservedRunningTime="2026-10-30 18:32:13.806710999 +0000 UTC m=+2030.631105290" Oct 30 18:32:14.365612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:14.365579 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:15.193993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:15.193957 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:16.369689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:16.369647 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:18.390655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:18.390605 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:19.785523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:19.785495 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:32:20.448674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:20.448638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:22.259886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:22.259852 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:22.763373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:22.763347 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 18:32:24.097024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:24.096989 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.089548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:26.089513 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.403746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:26.403715 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.848705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:26.848624 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:27.398755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:27.398724 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:27.523859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:27.519574 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:28.384126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:28.384088 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:30.368285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.368251 4582 scope.go:122] "RemoveContainer" containerID="cd7132eccf2952d0db631f94b611864b5d90f0e18d24801347c564eaffa7312c" Oct 30 18:32:30.378872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.378848 4582 scope.go:122] "RemoveContainer" containerID="96ea05b2652f871411890bdd46e447bf900707bf846dc5592c0ac60a16036c53" Oct 30 18:32:30.655075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.655004 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh"] Oct 30 18:32:30.661288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.661270 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.680709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.680686 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh"] Oct 30 18:32:30.684759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.684739 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-09ef1d-serving-cert\"" Oct 30 18:32:30.768738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.768707 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:30.774134 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.774113 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.774263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.774229 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.774354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.774340 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.774423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.774408 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmgp\" (UniqueName: \"kubernetes.io/projected/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-kube-api-access-kqmgp\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.875866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.875833 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmgp\" (UniqueName: \"kubernetes.io/projected/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-kube-api-access-kqmgp\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.876066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.875932 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.876066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.876052 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.876191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.876173 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.876696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.876670 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:30.877073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.877051 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.878639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.878619 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.878726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.878659 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.893145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.893122 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmgp\" (UniqueName: \"kubernetes.io/projected/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-kube-api-access-kqmgp\") pod \"maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.976186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.976096 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:30.994046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:30.994004 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:31.283270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:31.283241 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh"] Oct 30 18:32:31.284780 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:32:31.284755 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc76d911_27d1_4fa7_b986_47ccdb9b6c43.slice/crio-e28ad65d44f04727fdce5406eaee76ff58b35b84d32ff1bdcaf6d6cf0822fc7b WatchSource:0}: Error finding container e28ad65d44f04727fdce5406eaee76ff58b35b84d32ff1bdcaf6d6cf0822fc7b: Status 404 returned error can't find the container with id e28ad65d44f04727fdce5406eaee76ff58b35b84d32ff1bdcaf6d6cf0822fc7b Oct 30 18:32:31.917111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:31.917072 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" event={"ID":"cc76d911-27d1-4fa7-b986-47ccdb9b6c43","Type":"ContainerStarted","Data":"2489f10e6461cf2b76df1840713ac0ecd5f4c278ec0c5ee2ee2679baf05450a9"} Oct 30 18:32:31.917501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:31.917129 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" event={"ID":"cc76d911-27d1-4fa7-b986-47ccdb9b6c43","Type":"ContainerStarted","Data":"e28ad65d44f04727fdce5406eaee76ff58b35b84d32ff1bdcaf6d6cf0822fc7b"} Oct 30 18:32:31.917501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:31.917286 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:31.957947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:31.957904 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" podStartSLOduration=1.47722154 podStartE2EDuration="1.95789116s" totalImagesPullingTime="480.66962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:31.953386583 +0000 UTC m=+2048.777780876" watchObservedRunningTime="2026-10-30 18:32:31.95789116 +0000 UTC m=+2048.782285451" Oct 30 18:32:33.168196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:33.168100 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:35.061382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.061350 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq"] Oct 30 18:32:35.065054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.065032 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.100022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.099987 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-24b124-serving-cert\"" Oct 30 18:32:35.104191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.104159 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq"] Oct 30 18:32:35.138721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.138684 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:35.216055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.216023 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:35.238169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.238140 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95mm\" (UniqueName: \"kubernetes.io/projected/d57db115-c40a-49da-bc23-bb87da91713d-kube-api-access-n95mm\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.238332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.238190 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.238332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.238283 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d57db115-c40a-49da-bc23-bb87da91713d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.238430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.238415 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.321560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.321446 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:35.340054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.340022 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95mm\" (UniqueName: \"kubernetes.io/projected/d57db115-c40a-49da-bc23-bb87da91713d-kube-api-access-n95mm\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.340248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.340076 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.340248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.340116 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d57db115-c40a-49da-bc23-bb87da91713d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.340248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.340216 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.340929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.340904 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d57db115-c40a-49da-bc23-bb87da91713d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.342545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.342519 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.342640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.342553 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.356388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.356365 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95mm\" (UniqueName: \"kubernetes.io/projected/d57db115-c40a-49da-bc23-bb87da91713d-kube-api-access-n95mm\") pod \"maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.379518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.379492 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:35.682456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.682433 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq"] Oct 30 18:32:35.684287 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:32:35.684255 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57db115_c40a_49da_bc23_bb87da91713d.slice/crio-bfed43214b83295432761c5bae3c309a670fa83f087949aa45dc9e69d22e13b8 WatchSource:0}: Error finding container bfed43214b83295432761c5bae3c309a670fa83f087949aa45dc9e69d22e13b8: Status 404 returned error can't find the container with id bfed43214b83295432761c5bae3c309a670fa83f087949aa45dc9e69d22e13b8 Oct 30 18:32:35.952400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:35.952299 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" event={"ID":"d57db115-c40a-49da-bc23-bb87da91713d","Type":"ContainerStarted","Data":"bfed43214b83295432761c5bae3c309a670fa83f087949aa45dc9e69d22e13b8"} Oct 30 18:32:36.268271 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:32:36.267197 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:32:36.959038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:36.958998 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:36.962360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:36.962331 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" event={"ID":"d57db115-c40a-49da-bc23-bb87da91713d","Type":"ContainerStarted","Data":"11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23"} Oct 30 18:32:36.962522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:36.962407 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:36.998312 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:36.998262 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" podStartSLOduration=1.675767393 podStartE2EDuration="1.998245184s" totalImagesPullingTime="322.477791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:36.995703164 +0000 UTC m=+2053.820097456" watchObservedRunningTime="2026-10-30 18:32:36.998245184 +0000 UTC m=+2053.822639479" Oct 30 18:32:37.930454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:37.930426 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 18:32:38.852085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:38.852050 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:40.552508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:40.552471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:40.891127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:40.891091 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:41.046709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:41.046667 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:41.149740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:41.149656 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:41.254516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:41.254482 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:42.771853 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:42.771803 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:42.932514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:42.932483 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:42.975870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:42.975808 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 18:32:43.551244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:43.551205 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:43.672877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:43.672841 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:45.456735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.456696 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:45.523519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.523483 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b"] Oct 30 18:32:45.528012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.527985 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.548297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.548264 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b"] Oct 30 18:32:45.551849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.551808 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ab244c\"/\"openshift-service-ca.crt\"" Oct 30 18:32:45.552032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.551837 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ab244c\"/\"kube-root-ca.crt\"" Oct 30 18:32:45.554189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.554166 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ab244c\"/\"default-dockercfg-hlx64\"" Oct 30 18:32:45.554340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.554172 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ab244c\"/\"auth-test-model-ab244c-kserve-self-signed-certs\"" Oct 30 18:32:45.581869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.581844 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:45.607126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.607080 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tls-certs\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.607295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.607146 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tmp-dir\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.607295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.607276 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-home\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.607379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.607368 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-dshm\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.607416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.607404 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-model-cache\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.607535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.607519 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srq6m\" (UniqueName: \"kubernetes.io/projected/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-kube-api-access-srq6m\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.708637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.708545 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-dshm\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.708637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.708616 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-model-cache\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.708954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.708740 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srq6m\" (UniqueName: \"kubernetes.io/projected/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-kube-api-access-srq6m\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.708954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.708880 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tls-certs\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.708954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.708922 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tmp-dir\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.709132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.708986 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-home\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.709132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.709065 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-model-cache\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.709280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.709255 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tmp-dir\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.709379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.709360 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-home\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.711572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.711551 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-dshm\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.711793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.711775 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tls-certs\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.725367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.725346 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srq6m\" (UniqueName: \"kubernetes.io/projected/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-kube-api-access-srq6m\") pod \"auth-test-model-ab244c-kserve-6566c65cfc-lrq2b\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:45.844566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:45.844533 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:46.168455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:46.168411 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b"] Oct 30 18:32:46.170408 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:32:46.170376 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e701aa5_f87b_41f3_ba26_75ace56a6ba3.slice/crio-8e1c9b4349d02141fdb760a44e897461e2cb42e8c73127b1e48396e7ec018805 WatchSource:0}: Error finding container 8e1c9b4349d02141fdb760a44e897461e2cb42e8c73127b1e48396e7ec018805: Status 404 returned error can't find the container with id 8e1c9b4349d02141fdb760a44e897461e2cb42e8c73127b1e48396e7ec018805 Oct 30 18:32:46.695492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:46.695455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:47.051403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:47.051310 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" event={"ID":"8e701aa5-f87b-41f3-ba26-75ace56a6ba3","Type":"ContainerStarted","Data":"cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7"} Oct 30 18:32:47.051403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:47.051359 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" event={"ID":"8e701aa5-f87b-41f3-ba26-75ace56a6ba3","Type":"ContainerStarted","Data":"8e1c9b4349d02141fdb760a44e897461e2cb42e8c73127b1e48396e7ec018805"} Oct 30 18:32:47.090701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:47.090650 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" podStartSLOduration=2.090636083 podStartE2EDuration="2.090636083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:47.085861991 +0000 UTC m=+2063.910256282" watchObservedRunningTime="2026-10-30 18:32:47.090636083 +0000 UTC m=+2063.915030407" Oct 30 18:32:47.637608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:47.637577 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:47.844812 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:47.844783 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:49.767531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:49.767497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:51.755297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:51.755267 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.558478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:53.558419 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:55.442315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:55.442283 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:55.845385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:55.845298 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:55.857869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:55.857843 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:56.142012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:56.141980 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:32:57.315078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:57.315041 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:59.460720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:32:59.460689 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.826224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:01.826195 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.635036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:02.635002 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:03.028524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:03.028440 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4"] Oct 30 18:33:03.037212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:03.037176 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41300a1-9wgt4"] Oct 30 18:33:03.303452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:03.303376 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133eb005-69a1-4688-bff4-1cc0792e894a" path="/var/lib/kubelet/pods/133eb005-69a1-4688-bff4-1cc0792e894a/volumes" Oct 30 18:33:03.700451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:03.700422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.593075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:05.593041 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:06.228049 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:33:06.228005 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:33:06.366546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:06.366507 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:07.031755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:07.031714 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5"] Oct 30 18:33:07.038532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:07.038504 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea59957-rmnq5"] Oct 30 18:33:07.305282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:07.305188 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2d9f8a-3f69-402b-b329-303d4a7419fc" path="/var/lib/kubelet/pods/7b2d9f8a-3f69-402b-b329-303d4a7419fc/volumes" Oct 30 18:33:07.573363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:07.573271 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.174685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.174644 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.255304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.255272 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8"] Oct 30 18:33:09.261198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.261181 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.286438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.286410 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-d82e82-serving-cert\"" Oct 30 18:33:09.312872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.310536 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8"] Oct 30 18:33:09.321114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.321084 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.321261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.321129 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.321371 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.321343 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvd4c\" (UniqueName: \"kubernetes.io/projected/96bab029-b5b1-40d4-bb1d-df4a3461ee67-kube-api-access-dvd4c\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.321516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.321492 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bab029-b5b1-40d4-bb1d-df4a3461ee67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.349955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.349926 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.422899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.422864 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.422899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.422897 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.423138 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.423113 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvd4c\" (UniqueName: \"kubernetes.io/projected/96bab029-b5b1-40d4-bb1d-df4a3461ee67-kube-api-access-dvd4c\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.423270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.423242 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bab029-b5b1-40d4-bb1d-df4a3461ee67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.423861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.423809 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bab029-b5b1-40d4-bb1d-df4a3461ee67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.425436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.425378 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.425546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.425517 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.446346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.446324 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvd4c\" (UniqueName: \"kubernetes.io/projected/96bab029-b5b1-40d4-bb1d-df4a3461ee67-kube-api-access-dvd4c\") pod \"maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.537516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.537484 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.577457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.577417 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:09.883427 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:33:09.883384 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bab029_b5b1_40d4_bb1d_df4a3461ee67.slice/crio-0beb1ff5603c391df3566ef7f862b31e557e952402a32d3cf57d7acae6d51f22 WatchSource:0}: Error finding container 0beb1ff5603c391df3566ef7f862b31e557e952402a32d3cf57d7acae6d51f22: Status 404 returned error can't find the container with id 0beb1ff5603c391df3566ef7f862b31e557e952402a32d3cf57d7acae6d51f22 Oct 30 18:33:09.884244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:09.884214 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8"] Oct 30 18:33:10.249102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:10.249017 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" event={"ID":"96bab029-b5b1-40d4-bb1d-df4a3461ee67","Type":"ContainerStarted","Data":"0beb1ff5603c391df3566ef7f862b31e557e952402a32d3cf57d7acae6d51f22"} Oct 30 18:33:11.270699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:11.270644 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" event={"ID":"96bab029-b5b1-40d4-bb1d-df4a3461ee67","Type":"ContainerStarted","Data":"cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541"} Oct 30 18:33:11.271114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:11.270750 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:11.309568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:11.309519 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" podStartSLOduration=1.90225861 podStartE2EDuration="2.309505314s" totalImagesPullingTime="407.246704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:11.307133577 +0000 UTC m=+2088.131527884" watchObservedRunningTime="2026-10-30 18:33:11.309505314 +0000 UTC m=+2088.133899606" Oct 30 18:33:11.367790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:11.367756 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:12.426632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:12.426589 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:13.762032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:13.761998 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:15.308505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:15.308471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:15.719084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:15.719049 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.284791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:17.284764 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 18:33:17.674321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:17.674280 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:19.451242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:19.451150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:21.158445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:21.158415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:22.860103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:22.860073 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:24.625405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:24.625368 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.351674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:26.351643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:28.190091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:28.190058 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.414613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:30.414580 4582 scope.go:122] "RemoveContainer" containerID="e951a4bba81ec6596bcac37270d2a041c38234cb0fbc4ba199fb9ac2e081c443" Oct 30 18:33:30.776658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:30.776579 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.871519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:30.871477 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:32.772948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:32.772911 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:34.412019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.411985 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:34.748775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.748700 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w"] Oct 30 18:33:34.753680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.753651 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.783131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.783094 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w"] Oct 30 18:33:34.783352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.783311 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"default-dockercfg-xwr88\"" Oct 30 18:33:34.783462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.783421 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"kube-root-ca.crt\"" Oct 30 18:33:34.783614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.783592 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"openshift-service-ca.crt\"" Oct 30 18:33:34.783707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.783693 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"auth-test-model-675070-kserve-self-signed-certs\"" Oct 30 18:33:34.791494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.791464 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:34.869240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.869208 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-home\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.869400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.869254 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-model-cache\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.869400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.869329 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/479ec951-ca87-4d05-b1bc-c43696dbf965-tls-certs\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.869477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.869434 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-tmp-dir\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.869477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.869456 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xfr\" (UniqueName: \"kubernetes.io/projected/479ec951-ca87-4d05-b1bc-c43696dbf965-kube-api-access-t6xfr\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.869554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.869490 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-dshm\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.970677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.970648 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-home\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.970911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.970704 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-model-cache\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.970911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.970748 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/479ec951-ca87-4d05-b1bc-c43696dbf965-tls-certs\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.970911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.970871 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-tmp-dir\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.970911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.970901 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xfr\" (UniqueName: \"kubernetes.io/projected/479ec951-ca87-4d05-b1bc-c43696dbf965-kube-api-access-t6xfr\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.971155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.970947 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-dshm\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.971155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.971106 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-home\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.971274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.971183 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-model-cache\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.971274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.971229 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-tmp-dir\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.973186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.973165 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-dshm\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.973302 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.973264 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/479ec951-ca87-4d05-b1bc-c43696dbf965-tls-certs\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:34.987562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:34.987521 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xfr\" (UniqueName: \"kubernetes.io/projected/479ec951-ca87-4d05-b1bc-c43696dbf965-kube-api-access-t6xfr\") pod \"auth-test-model-675070-kserve-58b6877885-bwz7w\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:35.072389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:35.072297 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:35.399060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:35.399026 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w"] Oct 30 18:33:35.399300 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:33:35.399100 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod479ec951_ca87_4d05_b1bc_c43696dbf965.slice/crio-f647ca33a3396fa8537df47a20d33387e52851465eefafac5ca97a735123441d WatchSource:0}: Error finding container f647ca33a3396fa8537df47a20d33387e52851465eefafac5ca97a735123441d: Status 404 returned error can't find the container with id f647ca33a3396fa8537df47a20d33387e52851465eefafac5ca97a735123441d Oct 30 18:33:35.473621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:35.473583 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" event={"ID":"479ec951-ca87-4d05-b1bc-c43696dbf965","Type":"ContainerStarted","Data":"0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8"} Oct 30 18:33:35.474009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:35.473643 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" event={"ID":"479ec951-ca87-4d05-b1bc-c43696dbf965","Type":"ContainerStarted","Data":"f647ca33a3396fa8537df47a20d33387e52851465eefafac5ca97a735123441d"} Oct 30 18:33:35.508870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:35.508796 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" podStartSLOduration=1.508781776 podStartE2EDuration="1.508781776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:35.50553955 +0000 UTC m=+2112.329933862" watchObservedRunningTime="2026-10-30 18:33:35.508781776 +0000 UTC m=+2112.333176069" Oct 30 18:33:36.073031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:36.073001 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:36.226732 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:33:36.226694 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:33:36.263072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:36.263038 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:38.250583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:38.250547 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.317756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:40.317719 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.417095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:40.417058 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:41.962351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:41.962309 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:42.253798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:42.253714 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:44.074869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:44.074797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:45.072906 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:45.072873 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:45.086092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:45.086063 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:45.097588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:45.097563 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:45.575088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:45.575056 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:33:46.035855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:46.035797 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb"] Oct 30 18:33:46.040483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:46.040452 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a7a92-f8gnb"] Oct 30 18:33:46.081787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:46.081759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:47.301774 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:47.301738 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428d12ed-2e29-4d01-a648-bdb392bee574" path="/var/lib/kubelet/pods/428d12ed-2e29-4d01-a648-bdb392bee574/volumes" Oct 30 18:33:47.867732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:47.867696 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:49.790878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:49.790837 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:51.271525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:51.271492 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp"] Oct 30 18:33:51.304118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:51.304084 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn"] Oct 30 18:33:51.414625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:51.414580 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:51.524186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:51.524060 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:51.614131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:51.614096 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:53.371390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:53.371350 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:55.068972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:55.068937 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:56.272637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.272573 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" podUID="e0648deb-e81d-4a66-b358-574a744136b2" containerName="payload-processing" containerID="cri-o://6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2" gracePeriod=25 Oct 30 18:33:56.304411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.304371 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" podUID="0d808add-11cf-44dc-9476-f1c8f7138f63" containerName="payload-pre-processing" containerID="cri-o://acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e" gracePeriod=25 Oct 30 18:33:56.592332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.592301 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:33:56.631101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.631071 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:33:56.664982 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.664882 4582 generic.go:356] "Generic (PLEG): container finished" podID="0d808add-11cf-44dc-9476-f1c8f7138f63" containerID="acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e" exitCode=0 Oct 30 18:33:56.665223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.665136 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" Oct 30 18:33:56.665323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.665251 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" event={"ID":"0d808add-11cf-44dc-9476-f1c8f7138f63","Type":"ContainerDied","Data":"acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e"} Oct 30 18:33:56.665323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.665297 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn" event={"ID":"0d808add-11cf-44dc-9476-f1c8f7138f63","Type":"ContainerDied","Data":"5bf5334a265d4599a58e41aaa6d2b680f5f5b8f6f3b069139f86ab7df8ccfdb6"} Oct 30 18:33:56.665455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.665324 4582 scope.go:122] "RemoveContainer" containerID="acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e" Oct 30 18:33:56.668280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.668252 4582 generic.go:356] "Generic (PLEG): container finished" podID="e0648deb-e81d-4a66-b358-574a744136b2" containerID="6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2" exitCode=0 Oct 30 18:33:56.668486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.668466 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" event={"ID":"e0648deb-e81d-4a66-b358-574a744136b2","Type":"ContainerDied","Data":"6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2"} Oct 30 18:33:56.668581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.668519 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" event={"ID":"e0648deb-e81d-4a66-b358-574a744136b2","Type":"ContainerDied","Data":"d7809da07fa0156753c9f6faec97814f004ad87abe078bfe751a5d702f4cdd26"} Oct 30 18:33:56.668744 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.668595 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp" Oct 30 18:33:56.682175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.682146 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0648deb-e81d-4a66-b358-574a744136b2-kube-api-access-wnxxv\" (UniqueName: \"kubernetes.io/projected/e0648deb-e81d-4a66-b358-574a744136b2-kube-api-access-wnxxv\") pod \"e0648deb-e81d-4a66-b358-574a744136b2\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " Oct 30 18:33:56.682408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.682323 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0648deb-e81d-4a66-b358-574a744136b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0648deb-e81d-4a66-b358-574a744136b2-plugins-config-volume\") pod \"e0648deb-e81d-4a66-b358-574a744136b2\" (UID: \"e0648deb-e81d-4a66-b358-574a744136b2\") " Oct 30 18:33:56.682575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.682449 4582 scope.go:122] "RemoveContainer" containerID="acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e" Oct 30 18:33:56.683129 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:33:56.683089 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e\": container with ID starting with acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e not found: ID does not exist" containerID="acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e" Oct 30 18:33:56.683252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.683126 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e"} err="failed to get container status \"acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e\": rpc error: code = NotFound desc = could not find container \"acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e\": container with ID starting with acf514d8a268591c51b321624890b6a7eb1ded995160286e32af66bc88ddbd4e not found: ID does not exist" Oct 30 18:33:56.683252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.683150 4582 scope.go:122] "RemoveContainer" containerID="6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2" Oct 30 18:33:56.683388 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.683312 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0648deb-e81d-4a66-b358-574a744136b2-plugins-config-volume" pod "e0648deb-e81d-4a66-b358-574a744136b2" (UID: "e0648deb-e81d-4a66-b358-574a744136b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:56.684852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.684805 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0648deb-e81d-4a66-b358-574a744136b2-kube-api-access-wnxxv" pod "e0648deb-e81d-4a66-b358-574a744136b2" (UID: "e0648deb-e81d-4a66-b358-574a744136b2"). InnerVolumeSpecName "kube-api-access-wnxxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:56.696149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.696127 4582 scope.go:122] "RemoveContainer" containerID="6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2" Oct 30 18:33:56.696438 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:33:56.696417 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2\": container with ID starting with 6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2 not found: ID does not exist" containerID="6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2" Oct 30 18:33:56.696540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.696443 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2"} err="failed to get container status \"6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2\": rpc error: code = NotFound desc = could not find container \"6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2\": container with ID starting with 6e5eb6228029909126c90dbe72b00b1b651f4fc987b1c627dfbb1d402f0b8fd2 not found: ID does not exist" Oct 30 18:33:56.758120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.758083 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:56.784636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.784559 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d808add-11cf-44dc-9476-f1c8f7138f63-kube-api-access-7c4hf\" (UniqueName: \"kubernetes.io/projected/0d808add-11cf-44dc-9476-f1c8f7138f63-kube-api-access-7c4hf\") pod \"0d808add-11cf-44dc-9476-f1c8f7138f63\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " Oct 30 18:33:56.784636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.784600 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d808add-11cf-44dc-9476-f1c8f7138f63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d808add-11cf-44dc-9476-f1c8f7138f63-plugins-config-volume\") pod \"0d808add-11cf-44dc-9476-f1c8f7138f63\" (UID: \"0d808add-11cf-44dc-9476-f1c8f7138f63\") " Oct 30 18:33:56.784998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.784975 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d808add-11cf-44dc-9476-f1c8f7138f63-plugins-config-volume" pod "0d808add-11cf-44dc-9476-f1c8f7138f63" (UID: "0d808add-11cf-44dc-9476-f1c8f7138f63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:56.785188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.785167 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0648deb-e81d-4a66-b358-574a744136b2-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:33:56.785240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.785195 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d808add-11cf-44dc-9476-f1c8f7138f63-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:33:56.785240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.785205 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnxxv\" (UniqueName: \"kubernetes.io/projected/e0648deb-e81d-4a66-b358-574a744136b2-kube-api-access-wnxxv\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:33:56.786656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.786633 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d808add-11cf-44dc-9476-f1c8f7138f63-kube-api-access-7c4hf" pod "0d808add-11cf-44dc-9476-f1c8f7138f63" (UID: "0d808add-11cf-44dc-9476-f1c8f7138f63"). InnerVolumeSpecName "kube-api-access-7c4hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:56.886474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:56.886442 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4hf\" (UniqueName: \"kubernetes.io/projected/0d808add-11cf-44dc-9476-f1c8f7138f63-kube-api-access-7c4hf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:33:57.008467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:57.008438 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn"] Oct 30 18:33:57.017968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:57.017945 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1e0b31-67474fb659-w9xtn"] Oct 30 18:33:57.041782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:57.041505 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp"] Oct 30 18:33:57.045585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:57.045561 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1e0b31-786cbfd95b-cf8sp"] Oct 30 18:33:57.301461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:57.301400 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d808add-11cf-44dc-9476-f1c8f7138f63" path="/var/lib/kubelet/pods/0d808add-11cf-44dc-9476-f1c8f7138f63/volumes" Oct 30 18:33:57.301876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:57.301861 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0648deb-e81d-4a66-b358-574a744136b2" path="/var/lib/kubelet/pods/e0648deb-e81d-4a66-b358-574a744136b2/volumes" Oct 30 18:33:58.027225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:58.027193 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4"] Oct 30 18:33:58.036295 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:58.036270 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c79a68bc-wlkg4"] Oct 30 18:33:58.458212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:58.458175 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:59.301920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:33:59.301887 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26cc1c5-716e-4551-b853-9beed051b54c" path="/var/lib/kubelet/pods/e26cc1c5-716e-4551-b853-9beed051b54c/volumes" Oct 30 18:34:00.294154 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:00.294120 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:01.982572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:01.982540 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:03.666337 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:03.666303 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:05.428876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:05.428842 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:06.227148 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:34:06.227099 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:34:06.695235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:06.695187 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:06.957195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:06.957104 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f"] Oct 30 18:34:06.957429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:06.957405 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" podUID="6684ee56-e803-430a-8049-560bac7cf26a" containerName="maas-api" containerID="cri-o://e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945" gracePeriod=30 Oct 30 18:34:07.019518 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.019486 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:07.229682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.229640 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:34:07.328384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.328348 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6684ee56-e803-430a-8049-560bac7cf26a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee56-e803-430a-8049-560bac7cf26a-openshift-service-ca\") pod \"6684ee56-e803-430a-8049-560bac7cf26a\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " Oct 30 18:34:07.328557 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.328538 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-maas-api-tls\") pod \"6684ee56-e803-430a-8049-560bac7cf26a\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " Oct 30 18:34:07.328728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.328695 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6684ee56-e803-430a-8049-560bac7cf26a-kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/6684ee56-e803-430a-8049-560bac7cf26a-kube-api-access-bmgmb\") pod \"6684ee56-e803-430a-8049-560bac7cf26a\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " Oct 30 18:34:07.328935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.328901 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6684ee56-e803-430a-8049-560bac7cf26a-openshift-service-ca" pod "6684ee56-e803-430a-8049-560bac7cf26a" (UID: "6684ee56-e803-430a-8049-560bac7cf26a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:07.329227 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.329206 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-metrics-certs\") pod \"6684ee56-e803-430a-8049-560bac7cf26a\" (UID: \"6684ee56-e803-430a-8049-560bac7cf26a\") " Oct 30 18:34:07.330261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.330232 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6684ee56-e803-430a-8049-560bac7cf26a-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:34:07.332604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.332575 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-metrics-certs" pod "6684ee56-e803-430a-8049-560bac7cf26a" (UID: "6684ee56-e803-430a-8049-560bac7cf26a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:07.336108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.332880 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-maas-api-tls" pod "6684ee56-e803-430a-8049-560bac7cf26a" (UID: "6684ee56-e803-430a-8049-560bac7cf26a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:07.336505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.336482 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6684ee56-e803-430a-8049-560bac7cf26a-kube-api-access-bmgmb" pod "6684ee56-e803-430a-8049-560bac7cf26a" (UID: "6684ee56-e803-430a-8049-560bac7cf26a"). InnerVolumeSpecName "kube-api-access-bmgmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:07.432104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.432011 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:34:07.432104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.432047 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6684ee56-e803-430a-8049-560bac7cf26a-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:34:07.432104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.432062 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/6684ee56-e803-430a-8049-560bac7cf26a-kube-api-access-bmgmb\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:34:07.775560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.775525 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:07.776629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.776602 4582 generic.go:356] "Generic (PLEG): container finished" podID="6684ee56-e803-430a-8049-560bac7cf26a" containerID="e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945" exitCode=0 Oct 30 18:34:07.776742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.776718 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" Oct 30 18:34:07.776786 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.776743 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" event={"ID":"6684ee56-e803-430a-8049-560bac7cf26a","Type":"ContainerDied","Data":"e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945"} Oct 30 18:34:07.776856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.776800 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f" event={"ID":"6684ee56-e803-430a-8049-560bac7cf26a","Type":"ContainerDied","Data":"9d05b0041345c7925af887d79408018566de5f521864949b642f7de61e716499"} Oct 30 18:34:07.776856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.776838 4582 scope.go:122] "RemoveContainer" containerID="e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945" Oct 30 18:34:07.788957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.788936 4582 scope.go:122] "RemoveContainer" containerID="e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945" Oct 30 18:34:07.789278 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:34:07.789255 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945\": container with ID starting with e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945 not found: ID does not exist" containerID="e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945" Oct 30 18:34:07.789370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.789303 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945"} err="failed to get container status \"e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945\": rpc error: code = NotFound desc = could not find container \"e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945\": container with ID starting with e82de467007a937e7ed4a9d570efe5a5f4349ed25ceb4ac821d9740d82609945 not found: ID does not exist" Oct 30 18:34:07.824055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.824024 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f"] Oct 30 18:34:07.832991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.832967 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e0b31-8bc774c57-lp58f"] Oct 30 18:34:07.876073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.876042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:07.976485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:07.976457 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:08.055597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:08.055514 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:08.514552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:08.514518 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:09.030854 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:09.030805 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh"] Oct 30 18:34:09.040035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:09.040006 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc09254-8k7gh"] Oct 30 18:34:09.301849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:09.301747 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7689d0-a86d-49e2-8c4b-de9d3fe2192b" path="/var/lib/kubelet/pods/5c7689d0-a86d-49e2-8c4b-de9d3fe2192b/volumes" Oct 30 18:34:09.302246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:09.302231 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6684ee56-e803-430a-8049-560bac7cf26a" path="/var/lib/kubelet/pods/6684ee56-e803-430a-8049-560bac7cf26a/volumes" Oct 30 18:34:09.955185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:09.955150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:10.030934 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:10.030899 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29"] Oct 30 18:34:10.037584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:10.037542 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366f1a13-cgd29"] Oct 30 18:34:11.301349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:11.301308 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b6bb82-963d-4f3a-b2e9-fd8128354dec" path="/var/lib/kubelet/pods/81b6bb82-963d-4f3a-b2e9-fd8128354dec/volumes" Oct 30 18:34:11.647999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:11.647958 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:13.351175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:13.351143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:14.631092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.631059 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:14.726410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.726373 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj"] Oct 30 18:34:14.727411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727387 4582 cpu_manager.go:414] "containerMap: removing container" podUID="6684ee56-e803-430a-8049-560bac7cf26a" containerName="maas-api" Oct 30 18:34:14.727531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727413 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6684ee56-e803-430a-8049-560bac7cf26a" containerName="maas-api" Oct 30 18:34:14.727531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727466 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0d808add-11cf-44dc-9476-f1c8f7138f63" containerName="payload-pre-processing" Oct 30 18:34:14.727531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727477 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d808add-11cf-44dc-9476-f1c8f7138f63" containerName="payload-pre-processing" Oct 30 18:34:14.727531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727507 4582 cpu_manager.go:414] "containerMap: removing container" podUID="e0648deb-e81d-4a66-b358-574a744136b2" containerName="payload-processing" Oct 30 18:34:14.727531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727516 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0648deb-e81d-4a66-b358-574a744136b2" containerName="payload-processing" Oct 30 18:34:14.727737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727685 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d808add-11cf-44dc-9476-f1c8f7138f63" containerName="payload-pre-processing" Oct 30 18:34:14.727737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727706 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0648deb-e81d-4a66-b358-574a744136b2" containerName="payload-processing" Oct 30 18:34:14.727737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.727723 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="6684ee56-e803-430a-8049-560bac7cf26a" containerName="maas-api" Oct 30 18:34:14.733905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.733885 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.740282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.740252 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:14.757696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.757670 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"openshift-service-ca.crt\"" Oct 30 18:34:14.757883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.757736 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"kube-root-ca.crt\"" Oct 30 18:34:14.757974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.757852 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"default-dockercfg-b2jfh\"" Oct 30 18:34:14.758261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.758244 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-09ef1d\"/\"auth-test-model-09ef1d-kserve-self-signed-certs\"" Oct 30 18:34:14.763363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.763339 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj"] Oct 30 18:34:14.832213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.832187 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-home\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.832363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.832216 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-model-cache\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.832363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.832263 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-dshm\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.832461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.832372 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvkt5\" (UniqueName: \"kubernetes.io/projected/cfa22325-098d-4e0b-a8c3-5c5633786b97-kube-api-access-wvkt5\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.832505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.832471 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-tmp-dir\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.832655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.832631 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa22325-098d-4e0b-a8c3-5c5633786b97-tls-certs\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.837444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.837411 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:14.934253 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934174 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-tmp-dir\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934254 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa22325-098d-4e0b-a8c3-5c5633786b97-tls-certs\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934374 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-home\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934404 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-model-cache\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934476 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-dshm\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934560 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvkt5\" (UniqueName: \"kubernetes.io/projected/cfa22325-098d-4e0b-a8c3-5c5633786b97-kube-api-access-wvkt5\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934657 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-tmp-dir\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934840 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-home\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.934942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.934898 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-model-cache\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.936585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.936562 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-dshm\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.936680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.936665 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa22325-098d-4e0b-a8c3-5c5633786b97-tls-certs\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.952675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.952647 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvkt5\" (UniqueName: \"kubernetes.io/projected/cfa22325-098d-4e0b-a8c3-5c5633786b97-kube-api-access-wvkt5\") pod \"auth-test-model-09ef1d-kserve-7796b6c49-lbhdj\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:14.955368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:14.955333 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:15.049040 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:15.049007 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:15.378042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:15.378014 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj"] Oct 30 18:34:15.379914 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:34:15.379887 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfa22325_098d_4e0b_a8c3_5c5633786b97.slice/crio-d64955982ab971cd21d658b2d6b8d5ef087ab07c17c143908950ca92ea5029ce WatchSource:0}: Error finding container d64955982ab971cd21d658b2d6b8d5ef087ab07c17c143908950ca92ea5029ce: Status 404 returned error can't find the container with id d64955982ab971cd21d658b2d6b8d5ef087ab07c17c143908950ca92ea5029ce Oct 30 18:34:15.476969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:15.476943 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:15.574505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:15.574458 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:15.850682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:15.850586 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" event={"ID":"cfa22325-098d-4e0b-a8c3-5c5633786b97","Type":"ContainerStarted","Data":"7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88"} Oct 30 18:34:15.850682 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:15.850634 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" event={"ID":"cfa22325-098d-4e0b-a8c3-5c5633786b97","Type":"ContainerStarted","Data":"d64955982ab971cd21d658b2d6b8d5ef087ab07c17c143908950ca92ea5029ce"} Oct 30 18:34:15.885974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:15.885934 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" podStartSLOduration=1.885922598 podStartE2EDuration="1.885922598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:15.88479754 +0000 UTC m=+2152.709191832" watchObservedRunningTime="2026-10-30 18:34:15.885922598 +0000 UTC m=+2152.710316890" Oct 30 18:34:16.049372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:16.049342 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:17.261390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:17.261357 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:18.954773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:18.954734 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:19.043775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:19.043728 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq"] Oct 30 18:34:19.050897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:19.050844 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea5e2b-cldnq"] Oct 30 18:34:19.301404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:19.301332 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292b9bf3-0be3-41e2-9c53-aeca7a904ee4" path="/var/lib/kubelet/pods/292b9bf3-0be3-41e2-9c53-aeca7a904ee4/volumes" Oct 30 18:34:20.745349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:20.745294 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:21.028441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:21.028367 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq"] Oct 30 18:34:21.037755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:21.037723 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde4a6e0-f9bgq"] Oct 30 18:34:21.301802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:21.301721 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540773a8-343a-4f34-afd4-88177b5999d4" path="/var/lib/kubelet/pods/540773a8-343a-4f34-afd4-88177b5999d4/volumes" Oct 30 18:34:22.545405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:22.545372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:24.260064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:24.260028 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:25.049607 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:25.049569 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:25.062460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:25.062434 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:25.580431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:25.580400 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:25.944000 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:25.943964 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:34:25.981677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:25.981628 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:26.076946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:26.076913 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:26.164758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:26.164725 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:27.779902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:27.779868 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:29.581683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:29.581650 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:29.689976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:29.689940 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:29.799831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:29.799788 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:30.449663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:30.449628 4582 scope.go:122] "RemoveContainer" containerID="3c559a7a6824ee0fc295c1bb3f253ad083b9d53c688a54fc01330f7a929c4eeb" Oct 30 18:34:30.463184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:30.463162 4582 scope.go:122] "RemoveContainer" containerID="248a7afb1d639d955affb1c80df685d47fe07986cb86121eb1ab91892d5edd11" Oct 30 18:34:30.473407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:30.473390 4582 scope.go:122] "RemoveContainer" containerID="24725b85f00d939264278248afb548651d91ad4a855b9f2067233c2a5b2ccd86" Oct 30 18:34:30.482571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:30.482554 4582 scope.go:122] "RemoveContainer" containerID="baddd0a0af25810c040c9a628b2e54c513750c755e0f223eec4e14ef9e48090e" Oct 30 18:34:30.491318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:30.491300 4582 scope.go:122] "RemoveContainer" containerID="3a7188bff5eabd8fb41e89b4dedc2d001d41023aeacfe0ab68e05ad5ab077e04" Oct 30 18:34:30.499941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:30.499920 4582 scope.go:122] "RemoveContainer" containerID="08d4b9891ff2b5603b241f3fe4b431e77dad1d9602865e8dd98ab5fb716bb795" Oct 30 18:34:30.509264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:30.509242 4582 scope.go:122] "RemoveContainer" containerID="0f4f8a50912b5041532001881d19d84286affed3a5ed2e69edb7c7e0a16a47a2" Oct 30 18:34:31.440883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:31.440848 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:31.550834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:31.550786 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:33.584792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:33.584758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:35.371461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:35.371425 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:36.226829 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:34:36.226780 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:34:37.170697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:37.170662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:38.035666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:38.035636 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh"] Oct 30 18:34:38.045307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:38.045275 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200da0b7-89fzh"] Oct 30 18:34:38.966590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:38.966557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:39.301635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:39.301560 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d" path="/var/lib/kubelet/pods/ca78c81e-4d94-41b8-a4e6-9a9dca65ae0d/volumes" Oct 30 18:34:40.243689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.243649 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w"] Oct 30 18:34:40.249712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.249691 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.271147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.271122 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d82e82\"/\"default-dockercfg-n86bq\"" Oct 30 18:34:40.271315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.271247 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d82e82\"/\"kube-root-ca.crt\"" Oct 30 18:34:40.271315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.271287 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-d82e82\"/\"openshift-service-ca.crt\"" Oct 30 18:34:40.271315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.271304 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-d82e82\"/\"auth-test-model-d82e82-kserve-self-signed-certs\"" Oct 30 18:34:40.280624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.280591 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w"] Oct 30 18:34:40.297538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.297494 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:40.333898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.333866 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-tmp-dir\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.334045 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.334002 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-dshm\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.334090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.334052 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37589391-0e85-444d-8403-9742ad05dd41-tls-certs\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.334247 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.334226 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-home\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.334338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.334319 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-model-cache\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.334410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.334391 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frpmn\" (UniqueName: \"kubernetes.io/projected/37589391-0e85-444d-8403-9742ad05dd41-kube-api-access-frpmn\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.435608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.435580 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37589391-0e85-444d-8403-9742ad05dd41-tls-certs\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.435748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.435707 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-home\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.435748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.435735 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-model-cache\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.435861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.435756 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frpmn\" (UniqueName: \"kubernetes.io/projected/37589391-0e85-444d-8403-9742ad05dd41-kube-api-access-frpmn\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.435861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.435812 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-tmp-dir\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.435981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.435951 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-dshm\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.436195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.436170 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-model-cache\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.436195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.436188 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-home\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.436332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.436272 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-tmp-dir\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.438013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.437993 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-dshm\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.438228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.438208 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37589391-0e85-444d-8403-9742ad05dd41-tls-certs\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.453112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.453089 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frpmn\" (UniqueName: \"kubernetes.io/projected/37589391-0e85-444d-8403-9742ad05dd41-kube-api-access-frpmn\") pod \"auth-test-model-d82e82-kserve-5b5bc5c78f-b548w\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.566524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.566194 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:40.875790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.875755 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w"] Oct 30 18:34:40.876450 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:34:40.876416 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37589391_0e85_444d_8403_9742ad05dd41.slice/crio-c43b145bb804558ceebcdd292f18eea6ee87836a93ea04d24c214b00533dc073 WatchSource:0}: Error finding container c43b145bb804558ceebcdd292f18eea6ee87836a93ea04d24c214b00533dc073: Status 404 returned error can't find the container with id c43b145bb804558ceebcdd292f18eea6ee87836a93ea04d24c214b00533dc073 Oct 30 18:34:40.993660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:40.993609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:41.066188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:41.066137 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" event={"ID":"37589391-0e85-444d-8403-9742ad05dd41","Type":"ContainerStarted","Data":"a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43"} Oct 30 18:34:41.066408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:41.066204 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" event={"ID":"37589391-0e85-444d-8403-9742ad05dd41","Type":"ContainerStarted","Data":"c43b145bb804558ceebcdd292f18eea6ee87836a93ea04d24c214b00533dc073"} Oct 30 18:34:41.103908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:41.103858 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" podStartSLOduration=1.103843853 podStartE2EDuration="1.103843853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:41.100503983 +0000 UTC m=+2177.924898326" watchObservedRunningTime="2026-10-30 18:34:41.103843853 +0000 UTC m=+2177.928238136" Oct 30 18:34:41.569084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:41.569046 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:42.029542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:42.029505 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5"] Oct 30 18:34:42.039740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:42.039703 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-228d5d05-2qzg5"] Oct 30 18:34:42.927448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:42.927414 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:43.303596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:43.303520 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f992480b-5bf3-4ad3-9f54-dd0839b3564e" path="/var/lib/kubelet/pods/f992480b-5bf3-4ad3-9f54-dd0839b3564e/volumes" Oct 30 18:34:44.573387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:44.573328 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:46.349217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:46.349182 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:48.083376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:48.083340 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:49.780701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:49.780619 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:50.569289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:50.569259 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:50.582095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:50.582067 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:51.159986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:51.159951 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:34:51.478731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:51.478597 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:52.029218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:52.029151 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj"] Oct 30 18:34:52.038191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:52.038162 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a9505ee-nlfsj"] Oct 30 18:34:53.204505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:53.204475 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:53.301636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:53.301605 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb20b4a-e59d-4cb5-a95b-2d008268dd98" path="/var/lib/kubelet/pods/abb20b4a-e59d-4cb5-a95b-2d008268dd98/volumes" Oct 30 18:34:54.998334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:54.998301 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r"] Oct 30 18:34:55.003142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.003123 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.029272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.029237 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-160bf8-serving-cert\"" Oct 30 18:34:55.038946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.038914 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r"] Oct 30 18:34:55.044933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.044900 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m"] Oct 30 18:34:55.047516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.047496 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfd6d6b-qgg4m"] Oct 30 18:34:55.079346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.079311 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:55.143755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.143724 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-maas-api-tls\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.143991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.143918 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-metrics-certs\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.143991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.143981 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgzfb\" (UniqueName: \"kubernetes.io/projected/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-kube-api-access-tgzfb\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.144126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.144060 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-openshift-service-ca\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.179476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.179436 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:55.245360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.245320 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgzfb\" (UniqueName: \"kubernetes.io/projected/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-kube-api-access-tgzfb\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.245575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.245374 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-openshift-service-ca\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.245575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.245490 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-maas-api-tls\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.245769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.245658 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-metrics-certs\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.246157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.246133 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-openshift-service-ca\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.248410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.248358 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-metrics-certs\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.248497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.248447 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-maas-api-tls\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.267313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.267283 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgzfb\" (UniqueName: \"kubernetes.io/projected/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-kube-api-access-tgzfb\") pod \"maas-api-e2e-explicit-160bf8-c5866fc55-t452r\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.302074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.302044 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1472e70f-57eb-4748-9465-5a7333fec068" path="/var/lib/kubelet/pods/1472e70f-57eb-4748-9465-5a7333fec068/volumes" Oct 30 18:34:55.317955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.317923 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:55.333238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.333182 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:55.621064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:55.621040 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r"] Oct 30 18:34:55.622893 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:34:55.622868 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5579ed7a_dbe7_44ac_9f2d_63df331c95e0.slice/crio-6ad10b6bde5599d7694aaac0890d1deee98606d2385d89a5bf23de3954d36857 WatchSource:0}: Error finding container 6ad10b6bde5599d7694aaac0890d1deee98606d2385d89a5bf23de3954d36857: Status 404 returned error can't find the container with id 6ad10b6bde5599d7694aaac0890d1deee98606d2385d89a5bf23de3954d36857 Oct 30 18:34:56.203512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:56.203478 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" event={"ID":"5579ed7a-dbe7-44ac-9f2d-63df331c95e0","Type":"ContainerStarted","Data":"7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26"} Oct 30 18:34:56.204014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:56.203531 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" event={"ID":"5579ed7a-dbe7-44ac-9f2d-63df331c95e0","Type":"ContainerStarted","Data":"6ad10b6bde5599d7694aaac0890d1deee98606d2385d89a5bf23de3954d36857"} Oct 30 18:34:56.204014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:56.203676 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:34:56.242153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:56.242094 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" podStartSLOduration=1.837282498 podStartE2EDuration="2.242076125s" totalImagesPullingTime="404.793627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:56.23736551 +0000 UTC m=+2193.061759829" watchObservedRunningTime="2026-10-30 18:34:56.242076125 +0000 UTC m=+2193.066470419" Oct 30 18:34:57.057727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:57.057657 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:58.699901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:34:58.699863 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:00.028770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:00.028739 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9"] Oct 30 18:35:00.038308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:00.038269 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde2b9d4-zbbk9"] Oct 30 18:35:00.462418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:00.462379 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:01.303149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:01.303111 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b1fbfd-2dfc-4617-81dc-9eb1ded2b795" path="/var/lib/kubelet/pods/25b1fbfd-2dfc-4617-81dc-9eb1ded2b795/volumes" Oct 30 18:35:02.218474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:02.218444 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:35:02.291810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:02.291772 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:04.085507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:04.085472 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:05.029444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:05.029408 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75"] Oct 30 18:35:05.040385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:05.040354 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f"] Oct 30 18:35:05.044620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:05.044591 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42535f1c-s7f75"] Oct 30 18:35:05.048176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:05.048149 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffbf2f7d-fpd9f"] Oct 30 18:35:05.300410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:05.300337 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2" path="/var/lib/kubelet/pods/9d2bdbbd-90b7-4fd3-a41d-5730550f1ef2/volumes" Oct 30 18:35:05.300794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:05.300780 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f192d012-e8bc-496b-a03c-5cfb8820f38d" path="/var/lib/kubelet/pods/f192d012-e8bc-496b-a03c-5cfb8820f38d/volumes" Oct 30 18:35:05.853645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:05.853608 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:06.225129 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:35:06.225097 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:35:07.558753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:07.558719 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:08.026552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:08.026511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:08.127549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:08.127495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:09.238679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:09.238640 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:10.887225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:10.887189 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:12.622449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:12.622418 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:13.029332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:13.029259 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d"] Oct 30 18:35:13.038495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:13.038460 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8767f707-rkq5d"] Oct 30 18:35:13.301791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:13.301713 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64af1e67-48a2-4102-ad6b-3775ceb9fe48" path="/var/lib/kubelet/pods/64af1e67-48a2-4102-ad6b-3775ceb9fe48/volumes" Oct 30 18:35:14.470042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:14.470001 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:15.601275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:15.601242 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:16.447884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:16.447854 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:18.041884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:18.041850 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg"] Oct 30 18:35:18.050345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:18.050315 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3110575-86xrg"] Oct 30 18:35:18.503756 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:18.503723 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:19.301632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:19.301601 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1" path="/var/lib/kubelet/pods/b9d1f3cd-0cd7-4583-9273-dfd4fdbdfac1/volumes" Oct 30 18:35:20.352006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:20.351975 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:22.055052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:22.055009 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:23.991712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:23.991673 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:25.624531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:25.624493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:26.483665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:26.483608 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:27.614978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:27.614935 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:28.490031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:28.489997 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:30.440529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.440475 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:30.585747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.585711 4582 scope.go:122] "RemoveContainer" containerID="d3110f73c8f79e2edb1a19fce4df41a1544dde099c0ee716723c0f492b8c56e9" Oct 30 18:35:30.596706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.596680 4582 scope.go:122] "RemoveContainer" containerID="dda6810e8ec532048e824565e75eb2f5bfeb06a205b6622d0c0d56d2223ec000" Oct 30 18:35:30.607879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.607653 4582 scope.go:122] "RemoveContainer" containerID="b56a535a6eb3d5c2f65ac790295a4f17ebd2c2975e9cf5cb668df439d984486e" Oct 30 18:35:30.620830 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.620795 4582 scope.go:122] "RemoveContainer" containerID="1f7ddb1cb24e5662ade39339d9da9104e47252b37a26f3148537887b20419c98" Oct 30 18:35:30.631030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.631005 4582 scope.go:122] "RemoveContainer" containerID="0e467a90975b76b6726a388428edcfa439eeb9343ef5abf5c294daf5e37f1925" Oct 30 18:35:30.640852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.640799 4582 scope.go:122] "RemoveContainer" containerID="e290614f9c3638f448262108587457ffe7aa15a9505c4570a580d40540ad7342" Oct 30 18:35:30.651713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:30.651691 4582 scope.go:122] "RemoveContainer" containerID="60bbeabca6570c2baf4e3ad48d830a0d52f078eb9a72e42535babc1e0d391365" Oct 30 18:35:32.268432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:32.268382 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:34.062800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:34.062760 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:35.766158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:35.766126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:36.225262 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:35:36.225229 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:35:37.546674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:37.546638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:39.388430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:39.388387 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:40.026075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:40.026037 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh"] Oct 30 18:35:40.034997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:40.034965 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3daa466a-24mhh"] Oct 30 18:35:41.026948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:41.026912 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj"] Oct 30 18:35:41.036098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:41.036070 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8cd7bca-rnzsj"] Oct 30 18:35:41.267171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:41.267142 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:41.302384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:41.302308 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e910e0c-b792-4da8-9f18-6e53081c8371" path="/var/lib/kubelet/pods/3e910e0c-b792-4da8-9f18-6e53081c8371/volumes" Oct 30 18:35:41.302751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:41.302737 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb307a6-8a01-4b64-999c-15986e6c6f82" path="/var/lib/kubelet/pods/5fb307a6-8a01-4b64-999c-15986e6c6f82/volumes" Oct 30 18:35:43.103274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:43.103229 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:44.868145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:44.868100 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:46.678128 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:46.678082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:48.028523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:48.028490 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89"] Oct 30 18:35:48.037716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:48.037690 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5978fbab-hxj89"] Oct 30 18:35:48.527857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:48.527787 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:49.302133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:49.302101 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b4fda4-5ce4-46db-bb11-2cefb6520a4e" path="/var/lib/kubelet/pods/68b4fda4-5ce4-46db-bb11-2cefb6520a4e/volumes" Oct 30 18:35:50.248297 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:50.248257 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:51.899572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:51.899538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:52.028035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:52.028000 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc"] Oct 30 18:35:52.036766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:52.036736 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6db7832-ncbsc"] Oct 30 18:35:53.276454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:53.276416 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:53.302195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:53.302158 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc094205-a88e-42fe-83db-3445bed8c279" path="/var/lib/kubelet/pods/fc094205-a88e-42fe-83db-3445bed8c279/volumes" Oct 30 18:35:53.579045 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:53.578956 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg"] Oct 30 18:35:53.579253 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:53.579215 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" containerID="cri-o://644e2da1447c05f27ac8be3db9980ac0ec53f1ffd0f372e08d8701c2878d89db" gracePeriod=30 Oct 30 18:35:53.885654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:53.885579 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:55.867431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:55.867390 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:57.726226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:57.726194 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:58.698650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.698622 4582 generic.go:356] "Generic (PLEG): container finished" podID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerID="644e2da1447c05f27ac8be3db9980ac0ec53f1ffd0f372e08d8701c2878d89db" exitCode=0 Oct 30 18:35:58.698766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.698692 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" event={"ID":"aa979efc-88a6-433f-897e-997eb4c3b11e","Type":"ContainerDied","Data":"644e2da1447c05f27ac8be3db9980ac0ec53f1ffd0f372e08d8701c2878d89db"} Oct 30 18:35:58.869068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.869042 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:35:58.948036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.947990 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-credential-socket\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948086 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-data\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948233 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948130 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa979efc-88a6-433f-897e-997eb4c3b11e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa979efc-88a6-433f-897e-997eb4c3b11e-istiod-ca-cert\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948254 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-kube-api-access-p4k9s\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-kube-api-access-p4k9s\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948279 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-credential-socket" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:58.948483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948424 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-socket\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948472 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-podinfo\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948507 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-envoy\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948558 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-certs\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.948700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.948599 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-token\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-token\") pod \"aa979efc-88a6-433f-897e-997eb4c3b11e\" (UID: \"aa979efc-88a6-433f-897e-997eb4c3b11e\") " Oct 30 18:35:58.949280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.949249 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:58.950008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.949712 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa979efc-88a6-433f-897e-997eb4c3b11e-istiod-ca-cert" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:58.950008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.949917 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-certs" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:58.950008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.949935 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-socket" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:58.950383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.950356 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-data" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:58.951834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.951509 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-kube-api-access-p4k9s" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "kube-api-access-p4k9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:58.951834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.951773 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-token" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:58.952193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.952162 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-envoy" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:58.952730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:58.952697 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-podinfo" pod "aa979efc-88a6-433f-897e-997eb4c3b11e" (UID: "aa979efc-88a6-433f-897e-997eb4c3b11e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:35:59.050872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050835 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.050872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050877 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa979efc-88a6-433f-897e-997eb4c3b11e-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.051081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050890 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4k9s\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-kube-api-access-p4k9s\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.051081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050899 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.051081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050908 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.051081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050917 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.051081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050925 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa979efc-88a6-433f-897e-997eb4c3b11e-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.051081 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.050933 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa979efc-88a6-433f-897e-997eb4c3b11e-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:35:59.558412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.558372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:59.724186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.724121 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" event={"ID":"aa979efc-88a6-433f-897e-997eb4c3b11e","Type":"ContainerDied","Data":"e29de418f3538ef3c51fa9c578c658c5e5a753b721b8cd376b7c95dc03526b8b"} Oct 30 18:35:59.724513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.724496 4582 scope.go:122] "RemoveContainer" containerID="644e2da1447c05f27ac8be3db9980ac0ec53f1ffd0f372e08d8701c2878d89db" Oct 30 18:35:59.724739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.724193 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg" Oct 30 18:35:59.780059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.780034 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg"] Oct 30 18:35:59.787463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:35:59.787437 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-zlrrg"] Oct 30 18:36:00.090283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:00.090205 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r"] Oct 30 18:36:00.100729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:00.100704 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c2477f-whc8r"] Oct 30 18:36:01.301199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:01.301166 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" path="/var/lib/kubelet/pods/aa979efc-88a6-433f-897e-997eb4c3b11e/volumes" Oct 30 18:36:01.301624 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:01.301609 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb28498-fa64-4247-a556-e8d79a41c5d8" path="/var/lib/kubelet/pods/ceb28498-fa64-4247-a556-e8d79a41c5d8/volumes" Oct 30 18:36:01.314384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:01.314349 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:02.026032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:02.025998 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc"] Oct 30 18:36:02.035169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:02.035143 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89197c7f-z5mmc"] Oct 30 18:36:03.068457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:03.068421 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:03.303365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:03.303330 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32086a5d-e428-4c95-be26-8aa59580733d" path="/var/lib/kubelet/pods/32086a5d-e428-4c95-be26-8aa59580733d/volumes" Oct 30 18:36:04.379591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:04.379549 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:04.482042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:04.482006 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:04.899304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:04.899270 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:06.225835 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:36:06.225784 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:36:06.753405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:06.753372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:08.627535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:08.627502 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp"] Oct 30 18:36:08.628120 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:08.627705 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" containerID="cri-o://f278688c1593cc97386baa3261f634adf7193786fe1c6a82fee2b2f82fe2cd28" gracePeriod=30 Oct 30 18:36:08.636746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:08.636715 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp"] Oct 30 18:36:08.636960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:08.636915 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" containerID="cri-o://2db9dfb0d38d4a0979851857f4298c4307580e738a85a1d6766506b50aa60187" gracePeriod=30 Oct 30 18:36:08.784613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:08.784569 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:10.461041 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:10.461007 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:12.154009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:12.153970 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:13.036486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.036450 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm"] Oct 30 18:36:13.045795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.045770 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dec613b-766zm"] Oct 30 18:36:13.302011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.301935 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363c20bc-90e6-45c8-9421-fa9d1e55c3f9" path="/var/lib/kubelet/pods/363c20bc-90e6-45c8-9421-fa9d1e55c3f9/volumes" Oct 30 18:36:13.815073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.814955 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:13.852133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.852097 4582 generic.go:356] "Generic (PLEG): container finished" podID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerID="2db9dfb0d38d4a0979851857f4298c4307580e738a85a1d6766506b50aa60187" exitCode=0 Oct 30 18:36:13.852288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.852172 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" event={"ID":"11ed87cc-00c3-47b9-b210-dc5a9a852487","Type":"ContainerDied","Data":"2db9dfb0d38d4a0979851857f4298c4307580e738a85a1d6766506b50aa60187"} Oct 30 18:36:13.853943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.853921 4582 generic.go:356] "Generic (PLEG): container finished" podID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerID="f278688c1593cc97386baa3261f634adf7193786fe1c6a82fee2b2f82fe2cd28" exitCode=0 Oct 30 18:36:13.854053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.853966 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" event={"ID":"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8","Type":"ContainerDied","Data":"f278688c1593cc97386baa3261f634adf7193786fe1c6a82fee2b2f82fe2cd28"} Oct 30 18:36:13.991421 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.991393 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:36:13.994961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:13.994943 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:36:14.070971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.070939 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-token\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-token\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071023 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istiod-ca-cert\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071057 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-credential-socket\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071090 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-socket\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071123 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-kube-api-access-d4djj\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-kube-api-access-d4djj\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071240 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-certs\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071278 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-data\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071317 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-envoy\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071504 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-podinfo\") pod \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\" (UID: \"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8\") " Oct 30 18:36:14.071608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071548 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-credential-socket" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.071782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071704 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-socket" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.071998 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-data" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.072095 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istiod-ca-cert" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.072168 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-certs" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.072448 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.072470 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.072484 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.072496 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.072614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.072510 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.073813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.073774 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-envoy" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.073813 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.073806 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-token" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:14.074668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.074643 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-kube-api-access-d4djj" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "kube-api-access-d4djj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:14.075285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.075264 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-podinfo" pod "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" (UID: "2e62f1ef-ed97-4426-aa3d-3e304f50d4d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:14.173550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173512 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-data\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.173728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173667 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-certs\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.173728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173713 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-envoy\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.173900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173769 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-podinfo\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.173900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173791 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-data" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.174025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173938 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-kube-api-access-k8bql\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-kube-api-access-k8bql\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.174025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173989 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-token\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-token\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.174025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.173988 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-certs" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.174178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174051 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-socket\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.174178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174106 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ed87cc-00c3-47b9-b210-dc5a9a852487-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ed87cc-00c3-47b9-b210-dc5a9a852487-istiod-ca-cert\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.174178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174154 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-credential-socket\") pod \"11ed87cc-00c3-47b9-b210-dc5a9a852487\" (UID: \"11ed87cc-00c3-47b9-b210-dc5a9a852487\") " Oct 30 18:36:14.174589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174426 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-credential-socket" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.174589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174481 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ed87cc-00c3-47b9-b210-dc5a9a852487-istiod-ca-cert" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:14.174589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174558 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-socket" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.174968 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174948 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174973 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174987 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.174998 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.175011 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.175027 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4djj\" (UniqueName: \"kubernetes.io/projected/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8-kube-api-access-d4djj\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.175042 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ed87cc-00c3-47b9-b210-dc5a9a852487-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.175055 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.175266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.175068 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.176127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.176102 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-podinfo" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:14.176225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.176191 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-kube-api-access-k8bql" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "kube-api-access-k8bql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:14.176329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.176313 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-token" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:14.176389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.176370 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-envoy" pod "11ed87cc-00c3-47b9-b210-dc5a9a852487" (UID: "11ed87cc-00c3-47b9-b210-dc5a9a852487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:14.276814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.276744 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bql\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-kube-api-access-k8bql\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.276814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.276775 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.276814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.276785 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.276814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.276794 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ed87cc-00c3-47b9-b210-dc5a9a852487-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:14.863832 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.863797 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" Oct 30 18:36:14.864323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.863854 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp" event={"ID":"11ed87cc-00c3-47b9-b210-dc5a9a852487","Type":"ContainerDied","Data":"7ddf68c3a2eb8523cf24801cf5aecabddf0bdcebd4fd3b50ad6d0623252933d1"} Oct 30 18:36:14.864323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.863931 4582 scope.go:122] "RemoveContainer" containerID="2db9dfb0d38d4a0979851857f4298c4307580e738a85a1d6766506b50aa60187" Oct 30 18:36:14.865565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.865547 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" Oct 30 18:36:14.865565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.865565 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp" event={"ID":"2e62f1ef-ed97-4426-aa3d-3e304f50d4d8","Type":"ContainerDied","Data":"a39fe1cb0d3fca65ab8b3f53888f4d3847c99d65852c2b82547f6dcc756eb31e"} Oct 30 18:36:14.876679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.876657 4582 scope.go:122] "RemoveContainer" containerID="f278688c1593cc97386baa3261f634adf7193786fe1c6a82fee2b2f82fe2cd28" Oct 30 18:36:14.922383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.922326 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp"] Oct 30 18:36:14.932747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.932718 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-kqxfp"] Oct 30 18:36:14.960649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.960622 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp"] Oct 30 18:36:14.963970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:14.963929 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-jt8qp"] Oct 30 18:36:15.300738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:15.300663 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" path="/var/lib/kubelet/pods/11ed87cc-00c3-47b9-b210-dc5a9a852487/volumes" Oct 30 18:36:15.301169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:15.301156 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" path="/var/lib/kubelet/pods/2e62f1ef-ed97-4426-aa3d-3e304f50d4d8/volumes" Oct 30 18:36:15.668049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:15.668015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:15.803524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:15.803489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:17.078320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.078281 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r"] Oct 30 18:36:17.078789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.078529 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" podUID="5579ed7a-dbe7-44ac-9f2d-63df331c95e0" containerName="maas-api" containerID="cri-o://7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26" gracePeriod=30 Oct 30 18:36:17.128781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.128747 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:17.214326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.214288 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" podUID="5579ed7a-dbe7-44ac-9f2d-63df331c95e0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.231:8443/health\": dial tcp 10.132.0.231:8443: connect: connection refused" Oct 30 18:36:17.392452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.392422 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:36:17.520508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.520477 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-openshift-service-ca\") pod \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " Oct 30 18:36:17.520696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.520585 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-maas-api-tls\") pod \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " Oct 30 18:36:17.520696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.520655 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-kube-api-access-tgzfb\" (UniqueName: \"kubernetes.io/projected/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-kube-api-access-tgzfb\") pod \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " Oct 30 18:36:17.520696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.520690 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-metrics-certs\") pod \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\" (UID: \"5579ed7a-dbe7-44ac-9f2d-63df331c95e0\") " Oct 30 18:36:17.520986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.520955 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-openshift-service-ca" pod "5579ed7a-dbe7-44ac-9f2d-63df331c95e0" (UID: "5579ed7a-dbe7-44ac-9f2d-63df331c95e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:17.521158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.521133 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:17.522749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.522721 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-maas-api-tls" pod "5579ed7a-dbe7-44ac-9f2d-63df331c95e0" (UID: "5579ed7a-dbe7-44ac-9f2d-63df331c95e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:17.522886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.522765 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-metrics-certs" pod "5579ed7a-dbe7-44ac-9f2d-63df331c95e0" (UID: "5579ed7a-dbe7-44ac-9f2d-63df331c95e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:17.522886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.522788 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-kube-api-access-tgzfb" pod "5579ed7a-dbe7-44ac-9f2d-63df331c95e0" (UID: "5579ed7a-dbe7-44ac-9f2d-63df331c95e0"). InnerVolumeSpecName "kube-api-access-tgzfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:17.622509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.622477 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:17.622509 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.622509 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgzfb\" (UniqueName: \"kubernetes.io/projected/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-kube-api-access-tgzfb\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:17.622720 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.622518 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5579ed7a-dbe7-44ac-9f2d-63df331c95e0-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:17.894950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.894856 4582 generic.go:356] "Generic (PLEG): container finished" podID="5579ed7a-dbe7-44ac-9f2d-63df331c95e0" containerID="7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26" exitCode=0 Oct 30 18:36:17.894950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.894902 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" Oct 30 18:36:17.894950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.894936 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" event={"ID":"5579ed7a-dbe7-44ac-9f2d-63df331c95e0","Type":"ContainerDied","Data":"7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26"} Oct 30 18:36:17.895208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.894996 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r" event={"ID":"5579ed7a-dbe7-44ac-9f2d-63df331c95e0","Type":"ContainerDied","Data":"6ad10b6bde5599d7694aaac0890d1deee98606d2385d89a5bf23de3954d36857"} Oct 30 18:36:17.895208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.895024 4582 scope.go:122] "RemoveContainer" containerID="7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26" Oct 30 18:36:17.906514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.906498 4582 scope.go:122] "RemoveContainer" containerID="7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26" Oct 30 18:36:17.906788 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:36:17.906772 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26\": container with ID starting with 7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26 not found: ID does not exist" containerID="7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26" Oct 30 18:36:17.906866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.906797 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26"} err="failed to get container status \"7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26\": rpc error: code = NotFound desc = could not find container \"7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26\": container with ID starting with 7ce3e02316a0a3273a2502725f2f7b8b63967dffee2ba623b956fb2898547a26 not found: ID does not exist" Oct 30 18:36:17.913201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.913178 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:17.939344 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.939322 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r"] Oct 30 18:36:17.948235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:17.948213 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-160bf8-c5866fc55-t452r"] Oct 30 18:36:18.019714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:18.019687 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:18.115245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:18.115210 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:18.585961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:18.585887 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:19.301389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:19.301361 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5579ed7a-dbe7-44ac-9f2d-63df331c95e0" path="/var/lib/kubelet/pods/5579ed7a-dbe7-44ac-9f2d-63df331c95e0/volumes" Oct 30 18:36:19.930259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:19.930227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:21.584425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:21.584393 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:23.241869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:23.241805 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:23.462495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:23.462459 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th"] Oct 30 18:36:23.462706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:23.462667 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" containerID="cri-o://2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9" gracePeriod=30 Oct 30 18:36:24.788955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:24.788904 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:24.882164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:24.882105 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:25.069060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:25.068971 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:25.167956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:25.167916 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:26.822983 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:26.822949 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:28.590152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.590099 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:28.738006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.737946 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:36:28.864609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864576 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-socket\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.864772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864615 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-certs\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.864772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864661 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-data\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.864772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864688 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-podinfo\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.864772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864729 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istiod-ca-cert\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.864999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864905 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-credential-socket\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.864999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864933 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-socket" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:28.864999 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.864975 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-data" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:28.865229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865025 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-certs" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:28.865229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865066 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-envoy\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.865229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865108 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-kube-api-access-p2xf5\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-kube-api-access-p2xf5\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.865229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865138 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-credential-socket" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:28.865229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865154 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-token\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-token\") pod \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\" (UID: \"284b15a6-7a92-4062-8aa2-44b6c4ed9f79\") " Oct 30 18:36:28.865514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865259 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istiod-ca-cert" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:28.865696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865672 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.865846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865700 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.865846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865713 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.865846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865726 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.865846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.865738 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.867172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.867142 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-envoy" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:28.867272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.867225 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-kube-api-access-p2xf5" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "kube-api-access-p2xf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:28.867412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.867384 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-podinfo" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:28.867506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.867429 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-token" pod "284b15a6-7a92-4062-8aa2-44b6c4ed9f79" (UID: "284b15a6-7a92-4062-8aa2-44b6c4ed9f79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:28.967525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.967493 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.967525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.967522 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2xf5\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-kube-api-access-p2xf5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.967525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.967533 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.967752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.967543 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b15a6-7a92-4062-8aa2-44b6c4ed9f79-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:36:28.985385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.985350 4582 generic.go:356] "Generic (PLEG): container finished" podID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerID="2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9" exitCode=0 Oct 30 18:36:28.985791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.985768 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" event={"ID":"284b15a6-7a92-4062-8aa2-44b6c4ed9f79","Type":"ContainerDied","Data":"2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9"} Oct 30 18:36:28.986027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.986009 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" event={"ID":"284b15a6-7a92-4062-8aa2-44b6c4ed9f79","Type":"ContainerDied","Data":"0acd1534e92f8bacb2fa904856cdb7c46ba8982163573d3d468e6e5459e6f8df"} Oct 30 18:36:28.986156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.986144 4582 scope.go:122] "RemoveContainer" containerID="2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9" Oct 30 18:36:28.986512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:28.986498 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th" Oct 30 18:36:29.002627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:29.002604 4582 scope.go:122] "RemoveContainer" containerID="2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9" Oct 30 18:36:29.002933 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:36:29.002911 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9\": container with ID starting with 2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9 not found: ID does not exist" containerID="2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9" Oct 30 18:36:29.003017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:29.002944 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9"} err="failed to get container status \"2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9\": rpc error: code = NotFound desc = could not find container \"2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9\": container with ID starting with 2b1baecfd8073c68c3c964aca2cbfbad604fbc1f65cde370d7ed3c9c4a778cd9 not found: ID does not exist" Oct 30 18:36:29.033815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:29.033783 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th"] Oct 30 18:36:29.045735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:29.045703 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-rq5th"] Oct 30 18:36:29.301178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:29.301099 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" path="/var/lib/kubelet/pods/284b15a6-7a92-4062-8aa2-44b6c4ed9f79/volumes" Oct 30 18:36:30.407702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.407668 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:30.713882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.713781 4582 scope.go:122] "RemoveContainer" containerID="c5a441a49b6e7da48965a9ef45ae20546ee24af3af994454ee36d782529d8f32" Oct 30 18:36:30.725200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.725181 4582 scope.go:122] "RemoveContainer" containerID="0456fbec8a43245448332f100b33169241e9aafebdc3f168c9084e715918ee2f" Oct 30 18:36:30.734039 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.734018 4582 scope.go:122] "RemoveContainer" containerID="ac90b0ee654480bbb7591717b563d951f547527095e41bdee306fccf4c5f348c" Oct 30 18:36:30.742495 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.742480 4582 scope.go:122] "RemoveContainer" containerID="95f0e04fdde12cba9da1655d8d10f8858fda6f3890e14c688ca463147bd8bdad" Oct 30 18:36:30.751414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.751393 4582 scope.go:122] "RemoveContainer" containerID="de7535aea84192a80e4d276c8c68de9c4769a61082a2dc21f29951c0f008c1e9" Oct 30 18:36:30.759905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.759889 4582 scope.go:122] "RemoveContainer" containerID="a18ccc7785f9762aedf2b6d9784fe828b557bcfd12f6b79b08a3e208a4334623" Oct 30 18:36:30.768293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.768276 4582 scope.go:122] "RemoveContainer" containerID="c86403726d4cd8b61ad5265cfe6231d9b288a3587235be993131044275561bbb" Oct 30 18:36:30.776745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.776729 4582 scope.go:122] "RemoveContainer" containerID="db59485300c486498306cac60c8e3a3899bb48434fb1f4ce5a144833d895a77d" Oct 30 18:36:30.787757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.787736 4582 scope.go:122] "RemoveContainer" containerID="fb26283cb23c23e8d5d18de42104ccd9950a96c1770cb84a8952d05fd8db7a7f" Oct 30 18:36:30.798395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.798367 4582 scope.go:122] "RemoveContainer" containerID="742a1bed9ffc9efffe5dbaf83fb742d263b2978017b71707f75746e654fd3c94" Oct 30 18:36:30.807544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:30.807526 4582 scope.go:122] "RemoveContainer" containerID="883f8224df30895289801d3acd9aa28b2cb81a62b20a18edf3d46fbf565cee9e" Oct 30 18:36:32.249755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:32.249717 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:34.051850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:34.051803 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:35.718147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:35.718110 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:36.226127 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:36:36.226090 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:36:37.480124 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:37.480078 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:39.178736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:39.178702 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:40.942508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:40.942475 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:42.830106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:42.830078 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:44.533559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:44.533525 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:46.251560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:46.251527 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:48.046702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:48.046671 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:49.889429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:49.889395 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:51.686345 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:51.686314 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:53.401065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:53.401036 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:53.500877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:53.500846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:54.663407 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:54.663367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:54.925232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:54.925157 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:55.583266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:55.583232 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:57.470051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:57.470020 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:57.596254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:57.596224 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:58.604210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:58.604176 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:58.716222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:58.716186 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:59.444556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:36:59.444517 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:00.350530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:00.350500 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:00.457515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:00.457482 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:01.583608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:01.583557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:03.276752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:03.276719 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.193231 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:05.193190 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:06.225371 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:37:06.225336 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:37:07.120015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:07.119982 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:07.569544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:07.569457 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:08.982074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:08.982035 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:10.742237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:10.742201 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:12.508591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:12.508543 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:14.454280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:14.454242 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:15.783544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:15.783511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:16.208722 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:16.208683 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:17.993629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:17.993596 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:19.763060 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:19.763028 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:21.604259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:21.604224 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:23.385858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:23.385783 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:23.541899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:23.541858 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:23.685747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:23.685660 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:23.765520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:23.765476 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f"] Oct 30 18:37:23.765708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:23.765673 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" containerID="cri-o://273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658" gracePeriod=30 Oct 30 18:37:25.551140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:25.551107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:26.433552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:26.433519 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:27.486417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:27.486385 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:29.036974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.036911 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:37:29.069747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.069720 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07180983-3bdc-4785-a74f-0c1e22b42092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07180983-3bdc-4785-a74f-0c1e22b42092-istiod-ca-cert\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.069919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.069753 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-envoy\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.069919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.069844 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-certs\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.069919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.069894 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-kube-api-access-ctszz\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-kube-api-access-ctszz\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.070113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.069941 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-socket\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.070113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.069994 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-data\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.070113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070038 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-credential-socket\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.070113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070075 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-istio-token\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-istio-token\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.070351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070179 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07180983-3bdc-4785-a74f-0c1e22b42092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07180983-3bdc-4785-a74f-0c1e22b42092-istio-podinfo\") pod \"07180983-3bdc-4785-a74f-0c1e22b42092\" (UID: \"07180983-3bdc-4785-a74f-0c1e22b42092\") " Oct 30 18:37:29.070351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070188 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07180983-3bdc-4785-a74f-0c1e22b42092-istiod-ca-cert" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:29.070351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070243 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-certs" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.070351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070273 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-socket" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.070574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070371 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-data" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.070574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070406 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-credential-socket" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.070684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070669 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.070750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070689 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.070750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070706 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.070750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070719 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07180983-3bdc-4785-a74f-0c1e22b42092-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.070750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.070732 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.072243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.072215 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-envoy" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.072243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.072215 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-kube-api-access-ctszz" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "kube-api-access-ctszz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.072385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.072316 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07180983-3bdc-4785-a74f-0c1e22b42092-istio-podinfo" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:29.072385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.072352 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-istio-token" pod "07180983-3bdc-4785-a74f-0c1e22b42092" (UID: "07180983-3bdc-4785-a74f-0c1e22b42092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.171618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.171580 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07180983-3bdc-4785-a74f-0c1e22b42092-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.171618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.171620 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07180983-3bdc-4785-a74f-0c1e22b42092-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.171899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.171654 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctszz\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-kube-api-access-ctszz\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.171899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.171666 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07180983-3bdc-4785-a74f-0c1e22b42092-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.328083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.328047 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:29.383465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.383435 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b"] Oct 30 18:37:29.423272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.423243 4582 generic.go:356] "Generic (PLEG): container finished" podID="07180983-3bdc-4785-a74f-0c1e22b42092" containerID="273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658" exitCode=0 Oct 30 18:37:29.423460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.423292 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" Oct 30 18:37:29.423460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.423342 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" event={"ID":"07180983-3bdc-4785-a74f-0c1e22b42092","Type":"ContainerDied","Data":"273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658"} Oct 30 18:37:29.423460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.423394 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f" event={"ID":"07180983-3bdc-4785-a74f-0c1e22b42092","Type":"ContainerDied","Data":"602c5a745a8fd3492878b1dba97ee8ebb8277c1154f1227d3460a7b70c76b410"} Oct 30 18:37:29.423460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.423423 4582 scope.go:122] "RemoveContainer" containerID="273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658" Oct 30 18:37:29.433593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.433572 4582 scope.go:122] "RemoveContainer" containerID="273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658" Oct 30 18:37:29.433887 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:37:29.433866 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658\": container with ID starting with 273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658 not found: ID does not exist" containerID="273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658" Oct 30 18:37:29.433958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.433895 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658"} err="failed to get container status \"273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658\": rpc error: code = NotFound desc = could not find container \"273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658\": container with ID starting with 273678be85cf73658769e67f4ae04468732d968979934c4e28876d96b2868658 not found: ID does not exist" Oct 30 18:37:29.457023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.456945 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:29.471001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.470970 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f"] Oct 30 18:37:29.474442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:29.474415 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-slc6f"] Oct 30 18:37:30.920773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:30.920744 4582 scope.go:122] "RemoveContainer" containerID="d5c5d7cd6d942cb80262e91d2937077a2f55b31e1076e7e7fbdabedc1a71c6af" Oct 30 18:37:30.933317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:30.933299 4582 scope.go:122] "RemoveContainer" containerID="167a7cffea4ef28acb290871086487609af1f93c37bd25468ed97bb151d3d9aa" Oct 30 18:37:30.944522 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:30.944497 4582 scope.go:122] "RemoveContainer" containerID="087629a5c07c9969e498883fd64d06d5881c936534bb880c224f858cf05fc93f" Oct 30 18:37:30.953334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:30.953310 4582 scope.go:122] "RemoveContainer" containerID="e238cb8d1da25471df22b5bd098ef03f3b35c4e27d4506469cebe96dbb05c83b" Oct 30 18:37:30.961746 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:30.961723 4582 scope.go:122] "RemoveContainer" containerID="d5c5e23b2016bbf683fa43ec8941ff72d8620199ae689adb90e5bd3b359983b1" Oct 30 18:37:30.970596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:30.970576 4582 scope.go:122] "RemoveContainer" containerID="f452a49edadb6c54be429ae02aae6ac57e69938b5eabacb7192b93aaa6de71bd" Oct 30 18:37:30.978919 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:30.978873 4582 scope.go:122] "RemoveContainer" containerID="ae6969277383b3e39f83ad9c56aa77aee24f638fc75cf6316982ae7aa223b1b4" Oct 30 18:37:31.224105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:31.224016 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:31.302637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:31.302600 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" path="/var/lib/kubelet/pods/07180983-3bdc-4785-a74f-0c1e22b42092/volumes" Oct 30 18:37:33.098025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:33.097996 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:34.875789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:34.875744 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:36.225378 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:37:36.225339 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:37:36.657876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:36.657843 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:38.456736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:38.456703 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:40.261765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:40.261732 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:42.510334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:42.510297 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:43.635496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:43.635459 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:44.404930 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.404882 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" podUID="8e701aa5-f87b-41f3-ba26-75ace56a6ba3" containerName="main" containerID="cri-o://cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7" gracePeriod=45 Oct 30 18:37:44.675441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.675383 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:37:44.780701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.780668 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tmp-dir\") pod \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " Oct 30 18:37:44.780922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.780799 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-model-cache\") pod \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " Oct 30 18:37:44.780922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.780916 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tls-certs\") pod \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " Oct 30 18:37:44.781057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.780924 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tmp-dir" pod "8e701aa5-f87b-41f3-ba26-75ace56a6ba3" (UID: "8e701aa5-f87b-41f3-ba26-75ace56a6ba3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:44.781057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781025 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-model-cache" pod "8e701aa5-f87b-41f3-ba26-75ace56a6ba3" (UID: "8e701aa5-f87b-41f3-ba26-75ace56a6ba3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:44.781180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781056 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-dshm\") pod \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " Oct 30 18:37:44.781180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781090 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-kube-api-access-srq6m\" (UniqueName: \"kubernetes.io/projected/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-kube-api-access-srq6m\") pod \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " Oct 30 18:37:44.781180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781155 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-home\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-home\") pod \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\" (UID: \"8e701aa5-f87b-41f3-ba26-75ace56a6ba3\") " Oct 30 18:37:44.781438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781418 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-home" pod "8e701aa5-f87b-41f3-ba26-75ace56a6ba3" (UID: "8e701aa5-f87b-41f3-ba26-75ace56a6ba3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:44.781691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781674 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:44.781739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781693 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:44.781739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.781703 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:44.783280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.783257 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-kube-api-access-srq6m" pod "8e701aa5-f87b-41f3-ba26-75ace56a6ba3" (UID: "8e701aa5-f87b-41f3-ba26-75ace56a6ba3"). InnerVolumeSpecName "kube-api-access-srq6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:44.783438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.783422 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tls-certs" pod "8e701aa5-f87b-41f3-ba26-75ace56a6ba3" (UID: "8e701aa5-f87b-41f3-ba26-75ace56a6ba3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:44.783499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.783469 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-dshm" pod "8e701aa5-f87b-41f3-ba26-75ace56a6ba3" (UID: "8e701aa5-f87b-41f3-ba26-75ace56a6ba3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:44.789529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.789503 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:44.883587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.883550 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:44.883587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.883580 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:44.883587 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.883590 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srq6m\" (UniqueName: \"kubernetes.io/projected/8e701aa5-f87b-41f3-ba26-75ace56a6ba3-kube-api-access-srq6m\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:37:44.913719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:44.913681 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:45.528757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.528719 4582 generic.go:356] "Generic (PLEG): container finished" podID="8e701aa5-f87b-41f3-ba26-75ace56a6ba3" containerID="cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7" exitCode=0 Oct 30 18:37:45.528950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.528768 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" Oct 30 18:37:45.528950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.528801 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" event={"ID":"8e701aa5-f87b-41f3-ba26-75ace56a6ba3","Type":"ContainerDied","Data":"cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7"} Oct 30 18:37:45.528950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.528874 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b" event={"ID":"8e701aa5-f87b-41f3-ba26-75ace56a6ba3","Type":"ContainerDied","Data":"8e1c9b4349d02141fdb760a44e897461e2cb42e8c73127b1e48396e7ec018805"} Oct 30 18:37:45.528950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.528892 4582 scope.go:122] "RemoveContainer" containerID="cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7" Oct 30 18:37:45.538741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.538722 4582 scope.go:122] "RemoveContainer" containerID="cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7" Oct 30 18:37:45.539117 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:37:45.539098 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7\": container with ID starting with cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7 not found: ID does not exist" containerID="cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7" Oct 30 18:37:45.539193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.539130 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7"} err="failed to get container status \"cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7\": rpc error: code = NotFound desc = could not find container \"cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7\": container with ID starting with cd916d07bbc50f14ecb595c7e5e93b2e636b5012229277c2c87200c6fc98bbc7 not found: ID does not exist" Oct 30 18:37:45.577612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.577551 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b"] Oct 30 18:37:45.586133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:45.586106 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ab244c/auth-test-model-ab244c-kserve-6566c65cfc-lrq2b"] Oct 30 18:37:46.413404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:46.413367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:46.911129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:46.911086 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:47.058709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:47.058676 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:47.300724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:47.300650 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e701aa5-f87b-41f3-ba26-75ace56a6ba3" path="/var/lib/kubelet/pods/8e701aa5-f87b-41f3-ba26-75ace56a6ba3/volumes" Oct 30 18:37:48.805064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:48.804985 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:50.710276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:50.710245 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:52.615298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:52.615237 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:54.799172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:54.799140 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:56.806978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:56.806930 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:58.812649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:58.812616 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:59.904574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:37:59.904541 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:00.367055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:00.366968 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:00.987439 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:00.987392 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:01.427653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:01.427604 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:01.594124 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:01.594090 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:02.833908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:02.833873 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:02.940648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:02.940604 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:03.468285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:03.468225 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:05.112515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:05.112483 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:05.286868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:05.286793 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:05.628589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:05.628559 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:06.224415 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:38:06.224382 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:38:06.995211 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:06.995180 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:07.158097 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:07.158061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:07.909714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:07.909674 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:09.166878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:09.166845 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:09.894739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:09.894703 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:11.508709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:11.508674 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:11.991126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:11.991094 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:13.188577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:13.188539 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:14.043881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:14.043847 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w"] Oct 30 18:38:14.240790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:14.240754 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:14.693382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:14.693348 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:14.810971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:14.810933 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:15.464301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:15.464265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:15.656791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:15.656758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:15.951751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:15.951718 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:16.528642 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:16.528607 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:18.575808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:18.575762 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:20.712272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:20.712238 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:22.557839 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:22.557779 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:22.977488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:22.977455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:23.081636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:23.081605 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:24.304046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:24.304010 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:26.083865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:26.083805 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:27.120733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:27.120694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:27.243739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:27.243690 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:27.411953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:27.411786 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:28.063178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:28.063146 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:29.064949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.064882 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" podUID="37589391-0e85-444d-8403-9742ad05dd41" containerName="main" containerID="cri-o://a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43" gracePeriod=45 Oct 30 18:38:29.149683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.149650 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" podUID="37589391-0e85-444d-8403-9742ad05dd41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.230:8000/ready\": dial tcp 10.132.0.230:8000: connect: connection refused" Oct 30 18:38:29.351245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.351218 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:38:29.449993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.449959 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-home\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-home\") pod \"37589391-0e85-444d-8403-9742ad05dd41\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " Oct 30 18:38:29.450174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450014 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-dshm\") pod \"37589391-0e85-444d-8403-9742ad05dd41\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " Oct 30 18:38:29.450174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450128 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37589391-0e85-444d-8403-9742ad05dd41-tls-certs\" (UniqueName: \"kubernetes.io/secret/37589391-0e85-444d-8403-9742ad05dd41-tls-certs\") pod \"37589391-0e85-444d-8403-9742ad05dd41\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " Oct 30 18:38:29.450304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450221 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-model-cache\") pod \"37589391-0e85-444d-8403-9742ad05dd41\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " Oct 30 18:38:29.450304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450224 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-home" pod "37589391-0e85-444d-8403-9742ad05dd41" (UID: "37589391-0e85-444d-8403-9742ad05dd41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.450304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450261 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-tmp-dir\") pod \"37589391-0e85-444d-8403-9742ad05dd41\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " Oct 30 18:38:29.450480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450316 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37589391-0e85-444d-8403-9742ad05dd41-kube-api-access-frpmn\" (UniqueName: \"kubernetes.io/projected/37589391-0e85-444d-8403-9742ad05dd41-kube-api-access-frpmn\") pod \"37589391-0e85-444d-8403-9742ad05dd41\" (UID: \"37589391-0e85-444d-8403-9742ad05dd41\") " Oct 30 18:38:29.450559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450489 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-model-cache" pod "37589391-0e85-444d-8403-9742ad05dd41" (UID: "37589391-0e85-444d-8403-9742ad05dd41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.450559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450500 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-tmp-dir" pod "37589391-0e85-444d-8403-9742ad05dd41" (UID: "37589391-0e85-444d-8403-9742ad05dd41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.450955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450935 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.450955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450955 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.451157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.450965 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.452537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.452513 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37589391-0e85-444d-8403-9742ad05dd41-tls-certs" pod "37589391-0e85-444d-8403-9742ad05dd41" (UID: "37589391-0e85-444d-8403-9742ad05dd41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:29.452655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.452573 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37589391-0e85-444d-8403-9742ad05dd41-kube-api-access-frpmn" pod "37589391-0e85-444d-8403-9742ad05dd41" (UID: "37589391-0e85-444d-8403-9742ad05dd41"). InnerVolumeSpecName "kube-api-access-frpmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:29.452655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.452617 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-dshm" pod "37589391-0e85-444d-8403-9742ad05dd41" (UID: "37589391-0e85-444d-8403-9742ad05dd41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.552392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.552367 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37589391-0e85-444d-8403-9742ad05dd41-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.552392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.552391 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37589391-0e85-444d-8403-9742ad05dd41-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.552561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.552402 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frpmn\" (UniqueName: \"kubernetes.io/projected/37589391-0e85-444d-8403-9742ad05dd41-kube-api-access-frpmn\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.830317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.830282 4582 generic.go:356] "Generic (PLEG): container finished" podID="37589391-0e85-444d-8403-9742ad05dd41" containerID="a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43" exitCode=0 Oct 30 18:38:29.830498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.830327 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" Oct 30 18:38:29.830498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.830364 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" event={"ID":"37589391-0e85-444d-8403-9742ad05dd41","Type":"ContainerDied","Data":"a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43"} Oct 30 18:38:29.830498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.830412 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w" event={"ID":"37589391-0e85-444d-8403-9742ad05dd41","Type":"ContainerDied","Data":"c43b145bb804558ceebcdd292f18eea6ee87836a93ea04d24c214b00533dc073"} Oct 30 18:38:29.830498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.830434 4582 scope.go:122] "RemoveContainer" containerID="a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43" Oct 30 18:38:29.841421 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.841402 4582 scope.go:122] "RemoveContainer" containerID="a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43" Oct 30 18:38:29.841670 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:38:29.841652 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43\": container with ID starting with a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43 not found: ID does not exist" containerID="a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43" Oct 30 18:38:29.841726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.841678 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43"} err="failed to get container status \"a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43\": rpc error: code = NotFound desc = could not find container \"a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43\": container with ID starting with a0f48f1b5c68a422bf365bfdfc879a3a598768213cd3ae7aa336cb9b451c3f43 not found: ID does not exist" Oct 30 18:38:29.858488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.858452 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:29.887384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.887357 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w"] Oct 30 18:38:29.897581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:29.897546 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-d82e82/auth-test-model-d82e82-kserve-5b5bc5c78f-b548w"] Oct 30 18:38:31.300694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:31.300619 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37589391-0e85-444d-8403-9742ad05dd41" path="/var/lib/kubelet/pods/37589391-0e85-444d-8403-9742ad05dd41/volumes" Oct 30 18:38:31.555552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:31.555465 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:31.641444 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:31.641406 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:31.755301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:31.755268 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:33.591886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:33.591853 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:35.375034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:35.375002 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w"] Oct 30 18:38:35.405029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:35.405001 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:35.709550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:35.709470 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:36.226435 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:38:36.226398 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:38:37.450236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450196 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z"] Oct 30 18:38:37.450849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450803 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5579ed7a-dbe7-44ac-9f2d-63df331c95e0" containerName="maas-api" Oct 30 18:38:37.450849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450829 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5579ed7a-dbe7-44ac-9f2d-63df331c95e0" containerName="maas-api" Oct 30 18:38:37.450849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450842 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8e701aa5-f87b-41f3-ba26-75ace56a6ba3" containerName="main" Oct 30 18:38:37.450849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450848 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e701aa5-f87b-41f3-ba26-75ace56a6ba3" containerName="main" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450861 4582 cpu_manager.go:414] "containerMap: removing container" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450866 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450892 4582 cpu_manager.go:414] "containerMap: removing container" podUID="37589391-0e85-444d-8403-9742ad05dd41" containerName="main" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450897 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37589391-0e85-444d-8403-9742ad05dd41" containerName="main" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450909 4582 cpu_manager.go:414] "containerMap: removing container" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450914 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450922 4582 cpu_manager.go:414] "containerMap: removing container" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450927 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450942 4582 cpu_manager.go:414] "containerMap: removing container" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450947 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450953 4582 cpu_manager.go:414] "containerMap: removing container" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" Oct 30 18:38:37.451020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.450959 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451051 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ed87cc-00c3-47b9-b210-dc5a9a852487" containerName="istio-proxy" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451058 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="284b15a6-7a92-4062-8aa2-44b6c4ed9f79" containerName="istio-proxy" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451068 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="07180983-3bdc-4785-a74f-0c1e22b42092" containerName="istio-proxy" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451076 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa979efc-88a6-433f-897e-997eb4c3b11e" containerName="istio-proxy" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451082 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5579ed7a-dbe7-44ac-9f2d-63df331c95e0" containerName="maas-api" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451089 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e701aa5-f87b-41f3-ba26-75ace56a6ba3" containerName="main" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451095 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e62f1ef-ed97-4426-aa3d-3e304f50d4d8" containerName="istio-proxy" Oct 30 18:38:37.451391 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.451105 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="37589391-0e85-444d-8403-9742ad05dd41" containerName="main" Oct 30 18:38:37.454899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.454875 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.473331 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.473310 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z"] Oct 30 18:38:37.478911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.478882 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c095fa-serving-cert\"" Oct 30 18:38:37.547077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.547038 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.547241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.547100 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.547241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.547191 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqgh\" (UniqueName: \"kubernetes.io/projected/50e150fc-db48-4659-9c2f-4bf1f6d4318d-kube-api-access-qdqgh\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.547357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.547330 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e150fc-db48-4659-9c2f-4bf1f6d4318d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.554633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.554608 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:37.648625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.648599 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqgh\" (UniqueName: \"kubernetes.io/projected/50e150fc-db48-4659-9c2f-4bf1f6d4318d-kube-api-access-qdqgh\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.648785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.648707 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e150fc-db48-4659-9c2f-4bf1f6d4318d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.648888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.648869 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.648942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.648927 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.649073 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:38:37.649057 4582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-serving-cert: secret "maas-api-e2e-shared-b-w0-c095fa-serving-cert" not found Oct 30 18:38:37.649129 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:38:37.649119 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls podName:50e150fc-db48-4659-9c2f-4bf1f6d4318d nodeName:}" failed. No retries permitted until 2026-10-30 18:38:38.149099961 +0000 UTC m=+2414.973494231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls") pod "maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" (UID: "50e150fc-db48-4659-9c2f-4bf1f6d4318d") : secret "maas-api-e2e-shared-b-w0-c095fa-serving-cert" not found Oct 30 18:38:37.649591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.649561 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e150fc-db48-4659-9c2f-4bf1f6d4318d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.652339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.652199 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.670152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.670115 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:37.676007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.675977 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqgh\" (UniqueName: \"kubernetes.io/projected/50e150fc-db48-4659-9c2f-4bf1f6d4318d-kube-api-access-qdqgh\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:37.778781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:37.778688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:38.163336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:38.163283 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:38.167694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:38.167665 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:38.368189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:38.368159 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:38.699366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:38.699178 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z"] Oct 30 18:38:38.703301 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:38:38.703264 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e150fc_db48_4659_9c2f_4bf1f6d4318d.slice/crio-86fcdea900e72faa5ee2493650828e92302fe0e9e9083ed2c8a8cc1888a8a591 WatchSource:0}: Error finding container 86fcdea900e72faa5ee2493650828e92302fe0e9e9083ed2c8a8cc1888a8a591: Status 404 returned error can't find the container with id 86fcdea900e72faa5ee2493650828e92302fe0e9e9083ed2c8a8cc1888a8a591 Oct 30 18:38:38.705070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:38.705042 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:38:38.888881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:38.888846 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" event={"ID":"50e150fc-db48-4659-9c2f-4bf1f6d4318d","Type":"ContainerStarted","Data":"86fcdea900e72faa5ee2493650828e92302fe0e9e9083ed2c8a8cc1888a8a591"} Oct 30 18:38:39.690614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:39.690575 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:39.896327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:39.896292 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" event={"ID":"50e150fc-db48-4659-9c2f-4bf1f6d4318d","Type":"ContainerStarted","Data":"1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde"} Oct 30 18:38:39.896729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:39.896360 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:39.937209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:39.937165 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" podStartSLOduration=2.637460607 podStartE2EDuration="2.93715141s" totalImagesPullingTime="299.690803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:39.933949154 +0000 UTC m=+2416.758343476" watchObservedRunningTime="2026-10-30 18:38:39.93715141 +0000 UTC m=+2416.761545702" Oct 30 18:38:41.498194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:41.498157 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:41.904147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:41.904110 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:42.539198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:42.539163 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:42.699551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:42.699511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:43.599294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:43.599259 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:44.033552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:44.033517 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj"] Oct 30 18:38:44.073409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:44.073380 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:45.519798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:45.519763 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:45.908494 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:45.908466 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 18:38:46.651933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:46.651897 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:46.783749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:46.783711 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:47.979524 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:47.979491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:48.635993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:48.635935 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:48.751142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:48.751105 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:50.388259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.388220 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:50.395834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.395793 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" podUID="479ec951-ca87-4d05-b1bc-c43696dbf965" containerName="main" containerID="cri-o://0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8" gracePeriod=45 Oct 30 18:38:50.563859 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.563800 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" podUID="479ec951-ca87-4d05-b1bc-c43696dbf965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.228:8000/ready\": dial tcp 10.132.0.228:8000: connect: connection refused" Oct 30 18:38:50.665905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.665881 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:38:50.756955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.756910 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-model-cache\") pod \"479ec951-ca87-4d05-b1bc-c43696dbf965\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " Oct 30 18:38:50.756955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.756951 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479ec951-ca87-4d05-b1bc-c43696dbf965-kube-api-access-t6xfr\" (UniqueName: \"kubernetes.io/projected/479ec951-ca87-4d05-b1bc-c43696dbf965-kube-api-access-t6xfr\") pod \"479ec951-ca87-4d05-b1bc-c43696dbf965\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " Oct 30 18:38:50.757234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.757025 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479ec951-ca87-4d05-b1bc-c43696dbf965-tls-certs\" (UniqueName: \"kubernetes.io/secret/479ec951-ca87-4d05-b1bc-c43696dbf965-tls-certs\") pod \"479ec951-ca87-4d05-b1bc-c43696dbf965\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " Oct 30 18:38:50.757234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.757101 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-dshm\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-dshm\") pod \"479ec951-ca87-4d05-b1bc-c43696dbf965\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " Oct 30 18:38:50.757234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.757226 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-tmp-dir\") pod \"479ec951-ca87-4d05-b1bc-c43696dbf965\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " Oct 30 18:38:50.757417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.757264 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-home\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-home\") pod \"479ec951-ca87-4d05-b1bc-c43696dbf965\" (UID: \"479ec951-ca87-4d05-b1bc-c43696dbf965\") " Oct 30 18:38:50.758127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.758081 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-tmp-dir" pod "479ec951-ca87-4d05-b1bc-c43696dbf965" (UID: "479ec951-ca87-4d05-b1bc-c43696dbf965"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:50.758280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.758167 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-home" pod "479ec951-ca87-4d05-b1bc-c43696dbf965" (UID: "479ec951-ca87-4d05-b1bc-c43696dbf965"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:50.758396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.758372 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-model-cache" pod "479ec951-ca87-4d05-b1bc-c43696dbf965" (UID: "479ec951-ca87-4d05-b1bc-c43696dbf965"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:50.760519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.760497 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479ec951-ca87-4d05-b1bc-c43696dbf965-tls-certs" pod "479ec951-ca87-4d05-b1bc-c43696dbf965" (UID: "479ec951-ca87-4d05-b1bc-c43696dbf965"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:50.760626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.760551 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-dshm" pod "479ec951-ca87-4d05-b1bc-c43696dbf965" (UID: "479ec951-ca87-4d05-b1bc-c43696dbf965"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:50.760714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.760688 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479ec951-ca87-4d05-b1bc-c43696dbf965-kube-api-access-t6xfr" pod "479ec951-ca87-4d05-b1bc-c43696dbf965" (UID: "479ec951-ca87-4d05-b1bc-c43696dbf965"). InnerVolumeSpecName "kube-api-access-t6xfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:50.858533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.858506 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.858533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.858529 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.858533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.858539 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6xfr\" (UniqueName: \"kubernetes.io/projected/479ec951-ca87-4d05-b1bc-c43696dbf965-kube-api-access-t6xfr\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.858753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.858548 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/479ec951-ca87-4d05-b1bc-c43696dbf965-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.858753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.858556 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.858753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.858564 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/479ec951-ca87-4d05-b1bc-c43696dbf965-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.974648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.974569 4582 generic.go:356] "Generic (PLEG): container finished" podID="479ec951-ca87-4d05-b1bc-c43696dbf965" containerID="0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8" exitCode=0 Oct 30 18:38:50.974648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.974615 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" Oct 30 18:38:50.974869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.974655 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" event={"ID":"479ec951-ca87-4d05-b1bc-c43696dbf965","Type":"ContainerDied","Data":"0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8"} Oct 30 18:38:50.974869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.974700 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w" event={"ID":"479ec951-ca87-4d05-b1bc-c43696dbf965","Type":"ContainerDied","Data":"f647ca33a3396fa8537df47a20d33387e52851465eefafac5ca97a735123441d"} Oct 30 18:38:50.974869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.974717 4582 scope.go:122] "RemoveContainer" containerID="0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8" Oct 30 18:38:50.986013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.985994 4582 scope.go:122] "RemoveContainer" containerID="0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8" Oct 30 18:38:50.986270 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:38:50.986249 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8\": container with ID starting with 0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8 not found: ID does not exist" containerID="0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8" Oct 30 18:38:50.986342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:50.986282 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8"} err="failed to get container status \"0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8\": rpc error: code = NotFound desc = could not find container \"0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8\": container with ID starting with 0150d43fae2ebc07162d76922df8d9fd08c7dafbcfc581231c190aa6c97348f8 not found: ID does not exist" Oct 30 18:38:51.023570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:51.023540 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w"] Oct 30 18:38:51.028880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:51.028859 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/auth-test-model-675070-kserve-58b6877885-bwz7w"] Oct 30 18:38:51.300730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:51.300656 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479ec951-ca87-4d05-b1bc-c43696dbf965" path="/var/lib/kubelet/pods/479ec951-ca87-4d05-b1bc-c43696dbf965/volumes" Oct 30 18:38:52.343605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:52.343548 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:53.962162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:53.962128 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7"] Oct 30 18:38:53.962660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:53.962322 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" containerID="cri-o://dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa" gracePeriod=30 Oct 30 18:38:53.977116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:53.977087 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv"] Oct 30 18:38:53.977354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:53.977330 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" containerID="cri-o://b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb" gracePeriod=30 Oct 30 18:38:54.190298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:54.190266 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:55.689490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:55.689452 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:55.979240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:55.979162 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.219923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.219889 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.355901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.354889 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.869333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.869302 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h"] Oct 30 18:38:56.869963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.869947 4582 cpu_manager.go:414] "containerMap: removing container" podUID="479ec951-ca87-4d05-b1bc-c43696dbf965" containerName="main" Oct 30 18:38:56.869963 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.869962 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479ec951-ca87-4d05-b1bc-c43696dbf965" containerName="main" Oct 30 18:38:56.870090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.870080 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="479ec951-ca87-4d05-b1bc-c43696dbf965" containerName="main" Oct 30 18:38:56.875235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.875211 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:56.889367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.889341 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4c75d7\"/\"default-dockercfg-br55v\"" Oct 30 18:38:56.899080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.899056 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4c75d7\"/\"noten-model-4c75d7-kserve-self-signed-certs\"" Oct 30 18:38:56.899368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.899347 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4c75d7\"/\"kube-root-ca.crt\"" Oct 30 18:38:56.903935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.903913 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4c75d7\"/\"openshift-service-ca.crt\"" Oct 30 18:38:56.907127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.907106 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h"] Oct 30 18:38:56.941384 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.941356 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-home\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:56.941541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.941492 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-dshm\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:56.941623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.941607 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-model-cache\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:56.941694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.941682 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tls-certs\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:56.941749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.941730 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tmp-dir\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:56.941805 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.941797 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkr6w\" (UniqueName: \"kubernetes.io/projected/6ca1c78a-f95b-4580-8fc1-3cab65714a05-kube-api-access-tkr6w\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:56.975516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:56.975490 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:57.042652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.042619 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-model-cache\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.042775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.042678 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tls-certs\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.042775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.042732 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tmp-dir\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.042937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.042790 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkr6w\" (UniqueName: \"kubernetes.io/projected/6ca1c78a-f95b-4580-8fc1-3cab65714a05-kube-api-access-tkr6w\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.042937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.042833 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-home\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.042937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.042929 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-dshm\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.043114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.043102 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-model-cache\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.043242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.043219 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-home\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.043321 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.043307 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tmp-dir\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.045080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.045058 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-dshm\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.045323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.045306 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tls-certs\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.074831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.074800 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkr6w\" (UniqueName: \"kubernetes.io/projected/6ca1c78a-f95b-4580-8fc1-3cab65714a05-kube-api-access-tkr6w\") pod \"noten-model-4c75d7-kserve-867bf6654f-t2c9h\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.190514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.190438 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:57.534592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.534505 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h"] Oct 30 18:38:57.538106 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:38:57.538075 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ca1c78a_f95b_4580_8fc1_3cab65714a05.slice/crio-39878dd331fe504139e7906da29162f07a0a2e9a8ffd29f79af79d0da989988c WatchSource:0}: Error finding container 39878dd331fe504139e7906da29162f07a0a2e9a8ffd29f79af79d0da989988c: Status 404 returned error can't find the container with id 39878dd331fe504139e7906da29162f07a0a2e9a8ffd29f79af79d0da989988c Oct 30 18:38:57.849234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:57.849150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:58.023731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:58.023688 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" event={"ID":"6ca1c78a-f95b-4580-8fc1-3cab65714a05","Type":"ContainerStarted","Data":"c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906"} Oct 30 18:38:58.024197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:58.023741 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" event={"ID":"6ca1c78a-f95b-4580-8fc1-3cab65714a05","Type":"ContainerStarted","Data":"39878dd331fe504139e7906da29162f07a0a2e9a8ffd29f79af79d0da989988c"} Oct 30 18:38:58.060370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:58.060319 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" podStartSLOduration=2.060304942 podStartE2EDuration="2.060304942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:58.05658437 +0000 UTC m=+2434.880978662" watchObservedRunningTime="2026-10-30 18:38:58.060304942 +0000 UTC m=+2434.884699234" Oct 30 18:38:58.191606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:58.191578 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:38:59.054790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.054756 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" podUID="cfa22325-098d-4e0b-a8c3-5c5633786b97" containerName="main" containerID="cri-o://7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88" gracePeriod=45 Oct 30 18:38:59.349711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.349675 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:38:59.384814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.384787 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:38:59.391048 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.391026 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:38:59.473886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.473853 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-dshm\") pod \"cfa22325-098d-4e0b-a8c3-5c5633786b97\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " Oct 30 18:38:59.474086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.473982 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-kube-api-access-8c6lp\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-kube-api-access-8c6lp\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.474086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474035 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-socket\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.474086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474070 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-tmp-dir\") pod \"cfa22325-098d-4e0b-a8c3-5c5633786b97\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " Oct 30 18:38:59.474361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474341 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-tmp-dir" pod "cfa22325-098d-4e0b-a8c3-5c5633786b97" (UID: "cfa22325-098d-4e0b-a8c3-5c5633786b97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.474361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474351 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-kube-api-access-f2pf9\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-kube-api-access-f2pf9\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.474485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474455 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-token\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-token\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.474543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474519 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-home\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-home\") pod \"cfa22325-098d-4e0b-a8c3-5c5633786b97\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " Oct 30 18:38:59.474600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474566 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-socket\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.474658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474595 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-data\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.474715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474659 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-envoy\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.474715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474693 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-credential-socket\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.474851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474726 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa22325-098d-4e0b-a8c3-5c5633786b97-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa22325-098d-4e0b-a8c3-5c5633786b97-tls-certs\") pod \"cfa22325-098d-4e0b-a8c3-5c5633786b97\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " Oct 30 18:38:59.474851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474773 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-certs\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.474851 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474806 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-model-cache\") pod \"cfa22325-098d-4e0b-a8c3-5c5633786b97\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " Oct 30 18:38:59.475032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474864 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-data\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.475032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474914 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-podinfo\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.475032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474954 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-podinfo\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.475032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.474991 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-token\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-token\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.475032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.475023 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-credential-socket\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.475308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.475080 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa22325-098d-4e0b-a8c3-5c5633786b97-kube-api-access-wvkt5\" (UniqueName: \"kubernetes.io/projected/cfa22325-098d-4e0b-a8c3-5c5633786b97-kube-api-access-wvkt5\") pod \"cfa22325-098d-4e0b-a8c3-5c5633786b97\" (UID: \"cfa22325-098d-4e0b-a8c3-5c5633786b97\") " Oct 30 18:38:59.475308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.475114 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istiod-ca-cert\") pod \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\" (UID: \"b7c11a1e-43a9-46ff-ba67-72fe77cc0508\") " Oct 30 18:38:59.475308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.475172 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istiod-ca-cert\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.475308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.475254 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-certs\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.475308 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.475281 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-envoy\") pod \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\" (UID: \"2c107704-6d25-42cd-ab4d-b1c69eeb7759\") " Oct 30 18:38:59.476044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.476019 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.476249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.476229 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-kube-api-access-8c6lp" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "kube-api-access-8c6lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:59.476466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.476444 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-dshm" pod "cfa22325-098d-4e0b-a8c3-5c5633786b97" (UID: "cfa22325-098d-4e0b-a8c3-5c5633786b97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.476559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.476487 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-model-cache" pod "cfa22325-098d-4e0b-a8c3-5c5633786b97" (UID: "cfa22325-098d-4e0b-a8c3-5c5633786b97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.476686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.476668 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-credential-socket" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.476766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.476705 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-kube-api-access-f2pf9" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "kube-api-access-f2pf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:59.476993 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.476966 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-home" pod "cfa22325-098d-4e0b-a8c3-5c5633786b97" (UID: "cfa22325-098d-4e0b-a8c3-5c5633786b97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.478658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.477131 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-credential-socket" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.478658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.477448 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-data" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.478658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.477685 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-socket" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.478658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.478010 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-socket" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.478658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.478353 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istiod-ca-cert" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:59.479206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.479034 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-data" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.479206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.479057 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-certs" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.479206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.479187 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-certs" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.479383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.479300 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istiod-ca-cert" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:59.479559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.479532 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-token" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:59.479675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.479650 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-envoy" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.479898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.479874 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-podinfo" pod "b7c11a1e-43a9-46ff-ba67-72fe77cc0508" (UID: "b7c11a1e-43a9-46ff-ba67-72fe77cc0508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:59.480076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.480048 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-token" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:59.480076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.480059 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa22325-098d-4e0b-a8c3-5c5633786b97-tls-certs" pod "cfa22325-098d-4e0b-a8c3-5c5633786b97" (UID: "cfa22325-098d-4e0b-a8c3-5c5633786b97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:59.480194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.480097 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-envoy" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:59.481012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.480990 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-podinfo" pod "2c107704-6d25-42cd-ab4d-b1c69eeb7759" (UID: "2c107704-6d25-42cd-ab4d-b1c69eeb7759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:59.481738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.481720 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa22325-098d-4e0b-a8c3-5c5633786b97-kube-api-access-wvkt5" pod "cfa22325-098d-4e0b-a8c3-5c5633786b97" (UID: "cfa22325-098d-4e0b-a8c3-5c5633786b97"). InnerVolumeSpecName "kube-api-access-wvkt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577167 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6lp\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-kube-api-access-8c6lp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577198 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577208 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2pf9\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-kube-api-access-f2pf9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577217 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577226 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577233 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577241 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577249 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577258 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577268 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa22325-098d-4e0b-a8c3-5c5633786b97-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577276 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577286 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577293 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577301 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577308 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577315 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577323 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577331 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvkt5\" (UniqueName: \"kubernetes.io/projected/cfa22325-098d-4e0b-a8c3-5c5633786b97-kube-api-access-wvkt5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577339 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c11a1e-43a9-46ff-ba67-72fe77cc0508-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577346 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577354 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577361 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c107704-6d25-42cd-ab4d-b1c69eeb7759-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.577623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.577370 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa22325-098d-4e0b-a8c3-5c5633786b97-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:38:59.854254 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.854173 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:59.986179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:38:59.986139 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:00.037753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.037720 4582 generic.go:356] "Generic (PLEG): container finished" podID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerID="b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb" exitCode=0 Oct 30 18:39:00.037962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.037767 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" Oct 30 18:39:00.037962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.037800 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" event={"ID":"2c107704-6d25-42cd-ab4d-b1c69eeb7759","Type":"ContainerDied","Data":"b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb"} Oct 30 18:39:00.037962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.037869 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv" event={"ID":"2c107704-6d25-42cd-ab4d-b1c69eeb7759","Type":"ContainerDied","Data":"ab0e772b1bb5a68faa74d87cfde9b4ffd164aa5a5b8d19ac977b66c77d279d9e"} Oct 30 18:39:00.037962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.037899 4582 scope.go:122] "RemoveContainer" containerID="b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb" Oct 30 18:39:00.039339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.039317 4582 generic.go:356] "Generic (PLEG): container finished" podID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerID="dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa" exitCode=0 Oct 30 18:39:00.039452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.039366 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" Oct 30 18:39:00.039452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.039382 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" event={"ID":"b7c11a1e-43a9-46ff-ba67-72fe77cc0508","Type":"ContainerDied","Data":"dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa"} Oct 30 18:39:00.039452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.039436 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7" event={"ID":"b7c11a1e-43a9-46ff-ba67-72fe77cc0508","Type":"ContainerDied","Data":"d23df189b048a7b5e983ad94e514de7dc84f0991bb5c730694a4112bd138d30c"} Oct 30 18:39:00.040957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.040933 4582 generic.go:356] "Generic (PLEG): container finished" podID="cfa22325-098d-4e0b-a8c3-5c5633786b97" containerID="7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88" exitCode=0 Oct 30 18:39:00.041073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.040994 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" Oct 30 18:39:00.041073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.041019 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" event={"ID":"cfa22325-098d-4e0b-a8c3-5c5633786b97","Type":"ContainerDied","Data":"7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88"} Oct 30 18:39:00.041073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.041060 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj" event={"ID":"cfa22325-098d-4e0b-a8c3-5c5633786b97","Type":"ContainerDied","Data":"d64955982ab971cd21d658b2d6b8d5ef087ab07c17c143908950ca92ea5029ce"} Oct 30 18:39:00.053835 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.053804 4582 scope.go:122] "RemoveContainer" containerID="b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb" Oct 30 18:39:00.054435 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:39:00.054414 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb\": container with ID starting with b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb not found: ID does not exist" containerID="b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb" Oct 30 18:39:00.054560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.054447 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb"} err="failed to get container status \"b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb\": rpc error: code = NotFound desc = could not find container \"b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb\": container with ID starting with b59efce842841b07bb83d5c57a1841773357e732cf87daa1540e321d2e6991eb not found: ID does not exist" Oct 30 18:39:00.054560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.054468 4582 scope.go:122] "RemoveContainer" containerID="dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa" Oct 30 18:39:00.067198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.067175 4582 scope.go:122] "RemoveContainer" containerID="dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa" Oct 30 18:39:00.067624 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:39:00.067580 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa\": container with ID starting with dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa not found: ID does not exist" containerID="dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa" Oct 30 18:39:00.067771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.067632 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa"} err="failed to get container status \"dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa\": rpc error: code = NotFound desc = could not find container \"dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa\": container with ID starting with dee31eb4952c104ba413f2fb94f4f5201177703367c80e60e021b383e8a0f9fa not found: ID does not exist" Oct 30 18:39:00.067771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.067653 4582 scope.go:122] "RemoveContainer" containerID="7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88" Oct 30 18:39:00.087172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.087148 4582 scope.go:122] "RemoveContainer" containerID="7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88" Oct 30 18:39:00.087522 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:39:00.087499 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88\": container with ID starting with 7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88 not found: ID does not exist" containerID="7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88" Oct 30 18:39:00.087610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.087534 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88"} err="failed to get container status \"7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88\": rpc error: code = NotFound desc = could not find container \"7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88\": container with ID starting with 7d65bfcb238e886d55c03320e13aaa5bdcc3c0fc26e8e8656b41aa60e5b93a88 not found: ID does not exist" Oct 30 18:39:00.132667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.132636 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv"] Oct 30 18:39:00.142027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.141991 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-46hwv"] Oct 30 18:39:00.187201 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.187163 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7"] Oct 30 18:39:00.196214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.196186 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w98z7"] Oct 30 18:39:00.234454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.234431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:00.235935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.235907 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj"] Oct 30 18:39:00.240153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:00.240132 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-09ef1d/auth-test-model-09ef1d-kserve-7796b6c49-lbhdj"] Oct 30 18:39:01.301171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.301142 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" path="/var/lib/kubelet/pods/2c107704-6d25-42cd-ab4d-b1c69eeb7759/volumes" Oct 30 18:39:01.301629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.301614 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" path="/var/lib/kubelet/pods/b7c11a1e-43a9-46ff-ba67-72fe77cc0508/volumes" Oct 30 18:39:01.302062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.302047 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa22325-098d-4e0b-a8c3-5c5633786b97" path="/var/lib/kubelet/pods/cfa22325-098d-4e0b-a8c3-5c5633786b97/volumes" Oct 30 18:39:01.656479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.656442 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:01.757571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.757529 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:01.839084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839049 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg"] Oct 30 18:39:01.839741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839727 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cfa22325-098d-4e0b-a8c3-5c5633786b97" containerName="main" Oct 30 18:39:01.839788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839743 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa22325-098d-4e0b-a8c3-5c5633786b97" containerName="main" Oct 30 18:39:01.839788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839759 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" Oct 30 18:39:01.839788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839764 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" Oct 30 18:39:01.839788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839776 4582 cpu_manager.go:414] "containerMap: removing container" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" Oct 30 18:39:01.839788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839782 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" Oct 30 18:39:01.839985 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839977 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa22325-098d-4e0b-a8c3-5c5633786b97" containerName="main" Oct 30 18:39:01.840027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839988 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c11a1e-43a9-46ff-ba67-72fe77cc0508" containerName="istio-proxy" Oct 30 18:39:01.840027 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.839995 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c107704-6d25-42cd-ab4d-b1c69eeb7759" containerName="istio-proxy" Oct 30 18:39:01.845298 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.845280 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:01.864414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.864389 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg"] Oct 30 18:39:01.869488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.869462 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d1a330\"" Oct 30 18:39:01.869629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.869491 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d1a330-dockercfg-v4r4q\"" Oct 30 18:39:01.937816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.937718 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn"] Oct 30 18:39:01.943290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.943257 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:01.968748 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:01.968714 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn"] Oct 30 18:39:02.028752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.028710 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:02.029071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.029049 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jk4\" (UniqueName: \"kubernetes.io/projected/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-kube-api-access-x7jk4\") pod \"payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:02.130532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.130483 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jk4\" (UniqueName: \"kubernetes.io/projected/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-kube-api-access-x7jk4\") pod \"payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:02.130728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.130548 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa546d21-3cfb-473f-a3fd-470fa68ad607-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:02.130728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.130590 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:02.130728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.130686 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxxn\" (UniqueName: \"kubernetes.io/projected/fa546d21-3cfb-473f-a3fd-470fa68ad607-kube-api-access-2xxxn\") pod \"payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:02.131400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.131374 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:02.148895 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.148870 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jk4\" (UniqueName: \"kubernetes.io/projected/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-kube-api-access-x7jk4\") pod \"payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:02.158755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.158728 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:02.231789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.231757 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa546d21-3cfb-473f-a3fd-470fa68ad607-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:02.231975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.231856 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxxn\" (UniqueName: \"kubernetes.io/projected/fa546d21-3cfb-473f-a3fd-470fa68ad607-kube-api-access-2xxxn\") pod \"payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:02.232531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.232506 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa546d21-3cfb-473f-a3fd-470fa68ad607-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:02.248692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.248671 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxxn\" (UniqueName: \"kubernetes.io/projected/fa546d21-3cfb-473f-a3fd-470fa68ad607-kube-api-access-2xxxn\") pod \"payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:02.265185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.265110 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:02.315261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.315230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:02.456809 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.456784 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg"] Oct 30 18:39:02.459693 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:39:02.459662 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb32af3_0f8a_45e8_8e2b_3ffdd888c9cf.slice/crio-0d31b4be821c2b5e731862b13de8c558dddf96f8f7b388539216984bbbe18411 WatchSource:0}: Error finding container 0d31b4be821c2b5e731862b13de8c558dddf96f8f7b388539216984bbbe18411: Status 404 returned error can't find the container with id 0d31b4be821c2b5e731862b13de8c558dddf96f8f7b388539216984bbbe18411 Oct 30 18:39:02.570612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:02.570590 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn"] Oct 30 18:39:02.573388 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:39:02.573360 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa546d21_3cfb_473f_a3fd_470fa68ad607.slice/crio-6dc05e360f6c96ccd592c6449ebf1c8238851cfad58af62e8dde3dc0f7bf2523 WatchSource:0}: Error finding container 6dc05e360f6c96ccd592c6449ebf1c8238851cfad58af62e8dde3dc0f7bf2523: Status 404 returned error can't find the container with id 6dc05e360f6c96ccd592c6449ebf1c8238851cfad58af62e8dde3dc0f7bf2523 Oct 30 18:39:03.064133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:03.064097 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" event={"ID":"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf","Type":"ContainerStarted","Data":"27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242"} Oct 30 18:39:03.064329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:03.064189 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:03.064329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:03.064228 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" event={"ID":"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf","Type":"ContainerStarted","Data":"0d31b4be821c2b5e731862b13de8c558dddf96f8f7b388539216984bbbe18411"} Oct 30 18:39:03.065955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:03.065930 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" event={"ID":"fa546d21-3cfb-473f-a3fd-470fa68ad607","Type":"ContainerStarted","Data":"c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b"} Oct 30 18:39:03.066044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:03.065977 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" event={"ID":"fa546d21-3cfb-473f-a3fd-470fa68ad607","Type":"ContainerStarted","Data":"6dc05e360f6c96ccd592c6449ebf1c8238851cfad58af62e8dde3dc0f7bf2523"} Oct 30 18:39:03.066044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:03.066011 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:03.102358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:03.102319 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" podStartSLOduration=2.102307376 podStartE2EDuration="2.102307376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:03.098886657 +0000 UTC m=+2439.923280948" watchObservedRunningTime="2026-10-30 18:39:03.102307376 +0000 UTC m=+2439.926701717" Oct 30 18:39:04.592179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:04.592150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:05.596726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:05.596669 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" podStartSLOduration=4.596655805 podStartE2EDuration="4.596655805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:03.135358007 +0000 UTC m=+2439.959752300" watchObservedRunningTime="2026-10-30 18:39:05.596655805 +0000 UTC m=+2442.421050096" Oct 30 18:39:05.600492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:05.600464 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h"] Oct 30 18:39:05.641171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:05.641128 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:05.822145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:05.822105 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:06.164127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:06.164087 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:06.226761 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:39:06.226724 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:39:06.432997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:06.432910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:06.647449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:06.647389 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:08.507210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:08.507165 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:10.613653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:10.613621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.424516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:11.424480 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:12.336210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:12.336175 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:12.542441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:12.542409 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:14.076239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:14.076203 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 18:39:14.076628 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:14.076441 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 18:39:14.137289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:14.137254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:15.939189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:15.939150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:16.090300 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:16.090265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:17.745181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:17.745143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:19.427152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.427074 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg"] Oct 30 18:39:19.431025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.431008 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.448463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.448434 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg"] Oct 30 18:39:19.454770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.454744 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1c4bcd-serving-cert\"" Oct 30 18:39:19.499385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.499354 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:19.535114 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.535082 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce301cb9-9812-4f86-abd9-b1d41e8833ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.535339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.535127 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.535339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.535247 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.535339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.535296 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdlrw\" (UniqueName: \"kubernetes.io/projected/ce301cb9-9812-4f86-abd9-b1d41e8833ea-kube-api-access-sdlrw\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.581198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.581169 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:19.636902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.636876 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce301cb9-9812-4f86-abd9-b1d41e8833ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.637070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.636916 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.637070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.636982 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.637070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.637035 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdlrw\" (UniqueName: \"kubernetes.io/projected/ce301cb9-9812-4f86-abd9-b1d41e8833ea-kube-api-access-sdlrw\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.637537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.637518 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce301cb9-9812-4f86-abd9-b1d41e8833ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.639440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.639421 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.639504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.639480 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.653232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.653205 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdlrw\" (UniqueName: \"kubernetes.io/projected/ce301cb9-9812-4f86-abd9-b1d41e8833ea-kube-api-access-sdlrw\") pod \"maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:19.685716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.685647 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:19.744714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:19.744681 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:20.046635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:20.046554 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg"] Oct 30 18:39:20.050117 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:39:20.050090 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce301cb9_9812_4f86_abd9_b1d41e8833ea.slice/crio-068dcfb44a5c661f9fa99a249ed4cefd6ec45b911bb7dd56cfd68521d3f219fb WatchSource:0}: Error finding container 068dcfb44a5c661f9fa99a249ed4cefd6ec45b911bb7dd56cfd68521d3f219fb: Status 404 returned error can't find the container with id 068dcfb44a5c661f9fa99a249ed4cefd6ec45b911bb7dd56cfd68521d3f219fb Oct 30 18:39:20.178626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:20.178593 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" event={"ID":"ce301cb9-9812-4f86-abd9-b1d41e8833ea","Type":"ContainerStarted","Data":"068dcfb44a5c661f9fa99a249ed4cefd6ec45b911bb7dd56cfd68521d3f219fb"} Oct 30 18:39:20.621595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:20.621536 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" podUID="6ca1c78a-f95b-4580-8fc1-3cab65714a05" containerName="main" containerID="cri-o://c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906" gracePeriod=45 Oct 30 18:39:20.909412 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:20.909384 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:39:21.053397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053317 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-model-cache\") pod \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " Oct 30 18:39:21.053558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053505 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tls-certs\") pod \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " Oct 30 18:39:21.053558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053555 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca1c78a-f95b-4580-8fc1-3cab65714a05-kube-api-access-tkr6w\" (UniqueName: \"kubernetes.io/projected/6ca1c78a-f95b-4580-8fc1-3cab65714a05-kube-api-access-tkr6w\") pod \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " Oct 30 18:39:21.053690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053576 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tmp-dir\") pod \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " Oct 30 18:39:21.053690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053580 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-model-cache" pod "6ca1c78a-f95b-4580-8fc1-3cab65714a05" (UID: "6ca1c78a-f95b-4580-8fc1-3cab65714a05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:21.053690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053604 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-dshm\") pod \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " Oct 30 18:39:21.053690 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053624 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-home\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-home\") pod \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\" (UID: \"6ca1c78a-f95b-4580-8fc1-3cab65714a05\") " Oct 30 18:39:21.053947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.053775 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tmp-dir" pod "6ca1c78a-f95b-4580-8fc1-3cab65714a05" (UID: "6ca1c78a-f95b-4580-8fc1-3cab65714a05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:21.054036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.054000 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-home" pod "6ca1c78a-f95b-4580-8fc1-3cab65714a05" (UID: "6ca1c78a-f95b-4580-8fc1-3cab65714a05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:21.054162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.054146 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:39:21.054223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.054165 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:39:21.054223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.054175 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:39:21.055877 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.055853 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-dshm" pod "6ca1c78a-f95b-4580-8fc1-3cab65714a05" (UID: "6ca1c78a-f95b-4580-8fc1-3cab65714a05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:21.056239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.056222 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca1c78a-f95b-4580-8fc1-3cab65714a05-kube-api-access-tkr6w" pod "6ca1c78a-f95b-4580-8fc1-3cab65714a05" (UID: "6ca1c78a-f95b-4580-8fc1-3cab65714a05"). InnerVolumeSpecName "kube-api-access-tkr6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:21.056239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.056225 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tls-certs" pod "6ca1c78a-f95b-4580-8fc1-3cab65714a05" (UID: "6ca1c78a-f95b-4580-8fc1-3cab65714a05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:21.155208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.155168 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca1c78a-f95b-4580-8fc1-3cab65714a05-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:39:21.155208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.155203 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkr6w\" (UniqueName: \"kubernetes.io/projected/6ca1c78a-f95b-4580-8fc1-3cab65714a05-kube-api-access-tkr6w\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:39:21.155208 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.155216 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca1c78a-f95b-4580-8fc1-3cab65714a05-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:39:21.186689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.186652 4582 generic.go:356] "Generic (PLEG): container finished" podID="6ca1c78a-f95b-4580-8fc1-3cab65714a05" containerID="c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906" exitCode=0 Oct 30 18:39:21.186896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.186714 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" Oct 30 18:39:21.186896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.186740 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" event={"ID":"6ca1c78a-f95b-4580-8fc1-3cab65714a05","Type":"ContainerDied","Data":"c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906"} Oct 30 18:39:21.186896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.186782 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h" event={"ID":"6ca1c78a-f95b-4580-8fc1-3cab65714a05","Type":"ContainerDied","Data":"39878dd331fe504139e7906da29162f07a0a2e9a8ffd29f79af79d0da989988c"} Oct 30 18:39:21.186896 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.186802 4582 scope.go:122] "RemoveContainer" containerID="c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906" Oct 30 18:39:21.188622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.188595 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" event={"ID":"ce301cb9-9812-4f86-abd9-b1d41e8833ea","Type":"ContainerStarted","Data":"b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18"} Oct 30 18:39:21.188755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.188691 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:21.197635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.197619 4582 scope.go:122] "RemoveContainer" containerID="c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906" Oct 30 18:39:21.197998 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:39:21.197979 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906\": container with ID starting with c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906 not found: ID does not exist" containerID="c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906" Oct 30 18:39:21.198067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.198009 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906"} err="failed to get container status \"c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906\": rpc error: code = NotFound desc = could not find container \"c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906\": container with ID starting with c55b1c3d89f454f2563e4a76a01c771d76e6bfa0532446e3a08177801a8b7906 not found: ID does not exist" Oct 30 18:39:21.234891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.234847 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" podStartSLOduration=1.89664795 podStartE2EDuration="2.234831977s" totalImagesPullingTime="338.184027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:21.231867114 +0000 UTC m=+2458.056261406" watchObservedRunningTime="2026-10-30 18:39:21.234831977 +0000 UTC m=+2458.059226262" Oct 30 18:39:21.262637 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.262615 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h"] Oct 30 18:39:21.270178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.270157 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4c75d7/noten-model-4c75d7-kserve-867bf6654f-t2c9h"] Oct 30 18:39:21.301708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.301680 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca1c78a-f95b-4580-8fc1-3cab65714a05" path="/var/lib/kubelet/pods/6ca1c78a-f95b-4580-8fc1-3cab65714a05/volumes" Oct 30 18:39:21.538676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:21.538644 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:23.405749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:23.405718 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:23.949200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:23.949153 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:25.027695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:25.027650 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:25.128626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:25.128580 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:25.229662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:25.229614 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:26.668795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:26.668757 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:27.046088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:27.046011 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:27.204189 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:27.204161 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 18:39:28.881347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:28.881313 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:29.265735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:29.265647 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:29.389766 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:29.389738 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:30.986259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:30.986223 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:31.080332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:31.080300 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:31.184870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:31.184787 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:32.816708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:32.816674 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:34.643113 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.643081 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77"] Oct 30 18:39:34.643785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.643771 4582 cpu_manager.go:414] "containerMap: removing container" podUID="6ca1c78a-f95b-4580-8fc1-3cab65714a05" containerName="main" Oct 30 18:39:34.643874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.643786 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca1c78a-f95b-4580-8fc1-3cab65714a05" containerName="main" Oct 30 18:39:34.643935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.643924 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca1c78a-f95b-4580-8fc1-3cab65714a05" containerName="main" Oct 30 18:39:34.646458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.646442 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.670049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.670019 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d1a330-serving-cert\"" Oct 30 18:39:34.674283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.674247 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77"] Oct 30 18:39:34.720263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.720227 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.720423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.720383 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.720479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.720436 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97r9\" (UniqueName: \"kubernetes.io/projected/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-kube-api-access-b97r9\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.720646 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.720626 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-metrics-certs\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.730782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.730754 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:34.821987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.821955 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.822173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.822049 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.822173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.822090 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97r9\" (UniqueName: \"kubernetes.io/projected/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-kube-api-access-b97r9\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.822320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.822219 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-metrics-certs\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.822735 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.822708 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.824608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.824582 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.824852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.824795 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-metrics-certs\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.829638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.829616 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:34.838731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.838710 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97r9\" (UniqueName: \"kubernetes.io/projected/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-kube-api-access-b97r9\") pod \"maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:34.960223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:34.960129 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:35.262426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:35.262348 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77"] Oct 30 18:39:35.266038 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:39:35.266011 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6125b7_5d01_4ae2_aaa9_aa7746ab2a89.slice/crio-7db85849e0e092574b58f89ae57095160b74d56f5a125238234ba41322aa243c WatchSource:0}: Error finding container 7db85849e0e092574b58f89ae57095160b74d56f5a125238234ba41322aa243c: Status 404 returned error can't find the container with id 7db85849e0e092574b58f89ae57095160b74d56f5a125238234ba41322aa243c Oct 30 18:39:35.284962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:35.284932 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" event={"ID":"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89","Type":"ContainerStarted","Data":"7db85849e0e092574b58f89ae57095160b74d56f5a125238234ba41322aa243c"} Oct 30 18:39:36.225707 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:39:36.225672 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:39:36.293127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:36.293092 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" event={"ID":"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89","Type":"ContainerStarted","Data":"6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c"} Oct 30 18:39:36.293331 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:36.293258 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:36.329937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:36.329882 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" podStartSLOduration=1.9057724710000001 podStartE2EDuration="2.329867914s" totalImagesPullingTime="424.095443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:36.327051682 +0000 UTC m=+2473.151445985" watchObservedRunningTime="2026-10-30 18:39:36.329867914 +0000 UTC m=+2473.154262210" Oct 30 18:39:36.723598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:36.723564 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:38.420570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:38.420524 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:40.152342 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:40.152310 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:42.086785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:42.086745 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:42.305751 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:42.305721 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 18:39:43.939790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:43.939759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:45.849814 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:45.849778 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:47.695467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:47.695431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:48.229483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:48.229450 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:48.329396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:48.329356 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.140505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:49.140475 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.220848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:49.220802 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.505329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:49.505256 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.616014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:49.615977 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.722501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:49.722470 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.998057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:49.998018 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:50.404776 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:50.404741 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:50.510463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:50.510433 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:51.600806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:51.600776 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:53.542573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:53.542537 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:53.645499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:53.645465 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:53.761875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:53.761832 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:54.547016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:54.546985 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:54.652954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:54.652925 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:55.309405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.309370 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:55.405314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.405284 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:55.581411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.581298 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t"] Oct 30 18:39:55.586681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.586653 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.601143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.601111 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t"] Oct 30 18:39:55.617555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.617529 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-713dc8\"" Oct 30 18:39:55.617716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.617626 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-713dc8-dockercfg-v5rsw\"" Oct 30 18:39:55.682341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.682302 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wn4\" (UniqueName: \"kubernetes.io/projected/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-kube-api-access-d7wn4\") pod \"payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.682341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.682342 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.783347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.783315 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wn4\" (UniqueName: \"kubernetes.io/projected/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-kube-api-access-d7wn4\") pod \"payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.783517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.783357 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.784188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.784168 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.800250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.800215 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wn4\" (UniqueName: \"kubernetes.io/projected/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-kube-api-access-d7wn4\") pod \"payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.911319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.911287 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:55.923103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:55.923073 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:56.205899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:56.205813 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t"] Oct 30 18:39:56.209340 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:39:56.209310 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab77db3_946a_49b1_bfa1_444f9ecd00ab.slice/crio-b92e9c5dac2fd0dd1e6d4eb0ae7c095721b420dbd5a1ea398261bdcf4cb98658 WatchSource:0}: Error finding container b92e9c5dac2fd0dd1e6d4eb0ae7c095721b420dbd5a1ea398261bdcf4cb98658: Status 404 returned error can't find the container with id b92e9c5dac2fd0dd1e6d4eb0ae7c095721b420dbd5a1ea398261bdcf4cb98658 Oct 30 18:39:56.421570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:56.421536 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" event={"ID":"1ab77db3-946a-49b1-bfa1-444f9ecd00ab","Type":"ContainerStarted","Data":"651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b"} Oct 30 18:39:56.421791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:56.421596 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:39:56.421791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:56.421615 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" event={"ID":"1ab77db3-946a-49b1-bfa1-444f9ecd00ab","Type":"ContainerStarted","Data":"b92e9c5dac2fd0dd1e6d4eb0ae7c095721b420dbd5a1ea398261bdcf4cb98658"} Oct 30 18:39:56.455172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:56.455130 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" podStartSLOduration=1.455118532 podStartE2EDuration="1.455118532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:56.454661361 +0000 UTC m=+2493.279055653" watchObservedRunningTime="2026-10-30 18:39:56.455118532 +0000 UTC m=+2493.279512824" Oct 30 18:39:56.940493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:56.940463 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:57.037455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:57.037424 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:58.494990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:39:58.494950 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:00.280658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:00.280622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:02.040574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:02.040541 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:03.272666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:03.272625 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:04.066740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:04.066701 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:04.615514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:04.615479 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:05.084422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:05.084388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:05.219463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:05.219425 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:05.959729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:05.959696 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz"] Oct 30 18:40:05.963276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:05.963253 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:05.976918 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:05.976899 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-38e0db-serving-cert\"" Oct 30 18:40:05.982085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:05.982056 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz"] Oct 30 18:40:06.014398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.014371 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.014550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.014431 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xh7r\" (UniqueName: \"kubernetes.io/projected/af692e1e-ce02-464f-8ae5-637403b9a7b0-kube-api-access-8xh7r\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.014633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.014595 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-metrics-certs\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.014702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.014662 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af692e1e-ce02-464f-8ae5-637403b9a7b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.042105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.042073 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:06.115703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.115488 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.115703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.115565 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xh7r\" (UniqueName: \"kubernetes.io/projected/af692e1e-ce02-464f-8ae5-637403b9a7b0-kube-api-access-8xh7r\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.115703 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:40:06.115632 4582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-serving-cert: secret "maas-api-e2e-extmod-b-38e0db-serving-cert" not found Oct 30 18:40:06.115703 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:40:06.115706 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls podName:af692e1e-ce02-464f-8ae5-637403b9a7b0 nodeName:}" failed. No retries permitted until 2026-10-30 18:40:06.615688687 +0000 UTC m=+2503.440082980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls") pod "maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" (UID: "af692e1e-ce02-464f-8ae5-637403b9a7b0") : secret "maas-api-e2e-extmod-b-38e0db-serving-cert" not found Oct 30 18:40:06.116122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.115748 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-metrics-certs\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.116122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.115891 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af692e1e-ce02-464f-8ae5-637403b9a7b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.116602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.116577 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af692e1e-ce02-464f-8ae5-637403b9a7b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.117897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.117867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:06.118734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.118711 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-metrics-certs\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.132947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.132922 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xh7r\" (UniqueName: \"kubernetes.io/projected/af692e1e-ce02-464f-8ae5-637403b9a7b0-kube-api-access-8xh7r\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.228947 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:40:06.228862 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:40:06.261228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.261200 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:06.621360 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.621323 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.624085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.624054 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:06.876978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:06.876882 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:07.180874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:07.180775 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz"] Oct 30 18:40:07.210988 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:40:07.210944 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf692e1e_ce02_464f_8ae5_637403b9a7b0.slice/crio-b09dbee5ca8ae54f10c97cf7d3a6e81fc9b22e469d12a53b28e0ecb884c790b7 WatchSource:0}: Error finding container b09dbee5ca8ae54f10c97cf7d3a6e81fc9b22e469d12a53b28e0ecb884c790b7: Status 404 returned error can't find the container with id b09dbee5ca8ae54f10c97cf7d3a6e81fc9b22e469d12a53b28e0ecb884c790b7 Oct 30 18:40:07.430381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:07.430351 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 18:40:07.494132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:07.494047 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" event={"ID":"af692e1e-ce02-464f-8ae5-637403b9a7b0","Type":"ContainerStarted","Data":"b09dbee5ca8ae54f10c97cf7d3a6e81fc9b22e469d12a53b28e0ecb884c790b7"} Oct 30 18:40:08.085511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:08.085477 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:08.503427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:08.503393 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" event={"ID":"af692e1e-ce02-464f-8ae5-637403b9a7b0","Type":"ContainerStarted","Data":"a0f70aab0cd6233705b33bab3410bf7a62f0ee084de9d60fe5f30a640c28a6c9"} Oct 30 18:40:08.503915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:08.503474 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:08.538904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:08.538855 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" podStartSLOduration=3.2392846730000002 podStartE2EDuration="3.538838406s" totalImagesPullingTime="299.553733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:08.536997201 +0000 UTC m=+2505.361391535" watchObservedRunningTime="2026-10-30 18:40:08.538838406 +0000 UTC m=+2505.363232699" Oct 30 18:40:08.655491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:08.655454 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:08.757230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:08.757156 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:08.858415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:08.858386 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:10.309480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:10.309431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:10.465243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:10.465199 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:10.565604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:10.565497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:12.260710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:12.260630 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:14.054265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:14.054227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:14.516685 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:14.516652 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 18:40:15.946133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:15.946080 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:16.155338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:16.155303 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:16.689745 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:16.689698 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:18.002396 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:18.002361 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:18.244497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:18.244461 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:20.208754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:20.208683 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:22.070932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:22.070900 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:23.881990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:23.881928 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:25.561175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:25.561143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:27.236590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:27.236538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:28.733699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:28.733663 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:28.843728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:28.843697 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:29.022634 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:29.022555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:30.966962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:30.966928 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:32.650333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.650300 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz"] Oct 30 18:40:32.654773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.654756 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.671209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.671178 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz"] Oct 30 18:40:32.676484 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.676458 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-713dc8-serving-cert\"" Oct 30 18:40:32.722969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.722936 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:32.750573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.750533 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8907ec-250b-4c8e-aae5-2274127baa43-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.750769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.750590 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-metrics-certs\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.750769 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.750682 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkfn\" (UniqueName: \"kubernetes.io/projected/eb8907ec-250b-4c8e-aae5-2274127baa43-kube-api-access-wmkfn\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.750917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.750807 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-maas-api-tls\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.804386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.804322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:32.852588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.852553 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8907ec-250b-4c8e-aae5-2274127baa43-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.852767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.852627 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-metrics-certs\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.852767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.852652 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkfn\" (UniqueName: \"kubernetes.io/projected/eb8907ec-250b-4c8e-aae5-2274127baa43-kube-api-access-wmkfn\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.852767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.852710 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-maas-api-tls\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.853932 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.853904 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8907ec-250b-4c8e-aae5-2274127baa43-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.855852 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.855810 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-maas-api-tls\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.865382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.865350 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-metrics-certs\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.870196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.870137 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkfn\" (UniqueName: \"kubernetes.io/projected/eb8907ec-250b-4c8e-aae5-2274127baa43-kube-api-access-wmkfn\") pod \"maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:32.968892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:32.968786 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:33.268296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:33.268278 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz"] Oct 30 18:40:33.271459 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:40:33.271416 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8907ec_250b_4c8e_aae5_2274127baa43.slice/crio-4745e458f026686820977867f73d36199abd56dde3efbed3483212f5204ab2fa WatchSource:0}: Error finding container 4745e458f026686820977867f73d36199abd56dde3efbed3483212f5204ab2fa: Status 404 returned error can't find the container with id 4745e458f026686820977867f73d36199abd56dde3efbed3483212f5204ab2fa Oct 30 18:40:33.672352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:33.672314 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" event={"ID":"eb8907ec-250b-4c8e-aae5-2274127baa43","Type":"ContainerStarted","Data":"7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a"} Oct 30 18:40:33.672681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:33.672387 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" event={"ID":"eb8907ec-250b-4c8e-aae5-2274127baa43","Type":"ContainerStarted","Data":"4745e458f026686820977867f73d36199abd56dde3efbed3483212f5204ab2fa"} Oct 30 18:40:33.672681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:33.672534 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:33.710834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:33.710778 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" podStartSLOduration=1.395263485 podStartE2EDuration="1.710763778s" totalImagesPullingTime="315.500293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:33.707132173 +0000 UTC m=+2530.531526464" watchObservedRunningTime="2026-10-30 18:40:33.710763778 +0000 UTC m=+2530.535158072" Oct 30 18:40:34.538874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.538843 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9"] Oct 30 18:40:34.542354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.542334 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.556064 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.556042 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-861bb3-serving-cert\"" Oct 30 18:40:34.571405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.571373 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9"] Oct 30 18:40:34.610352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.610321 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:34.677078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.677043 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-maas-api-tls\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.677529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.677106 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-metrics-certs\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.677529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.677230 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0063e82-afb7-46df-9f43-324a55f24992-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.677529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.677281 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvm22\" (UniqueName: \"kubernetes.io/projected/f0063e82-afb7-46df-9f43-324a55f24992-kube-api-access-bvm22\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.692343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.692309 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:34.779098 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.779059 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-maas-api-tls\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.779319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.779249 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-metrics-certs\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.779404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.779329 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0063e82-afb7-46df-9f43-324a55f24992-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.779404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.779392 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvm22\" (UniqueName: \"kubernetes.io/projected/f0063e82-afb7-46df-9f43-324a55f24992-kube-api-access-bvm22\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.779960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.779932 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0063e82-afb7-46df-9f43-324a55f24992-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.781747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.781720 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-maas-api-tls\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.782004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.781983 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-metrics-certs\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.794975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.794912 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:34.795799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.795780 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvm22\" (UniqueName: \"kubernetes.io/projected/f0063e82-afb7-46df-9f43-324a55f24992-kube-api-access-bvm22\") pod \"maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:34.856544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:34.856517 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:35.153788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:35.153760 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9"] Oct 30 18:40:35.157039 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:40:35.157007 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0063e82_afb7_46df_9f43_324a55f24992.slice/crio-ac9fe2e995e0ad2ca3d1c443c96b40060f0d3c091e1409be758484bdc07af02d WatchSource:0}: Error finding container ac9fe2e995e0ad2ca3d1c443c96b40060f0d3c091e1409be758484bdc07af02d: Status 404 returned error can't find the container with id ac9fe2e995e0ad2ca3d1c443c96b40060f0d3c091e1409be758484bdc07af02d Oct 30 18:40:35.687245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:35.687212 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" event={"ID":"f0063e82-afb7-46df-9f43-324a55f24992","Type":"ContainerStarted","Data":"58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a"} Oct 30 18:40:35.687677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:35.687267 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" event={"ID":"f0063e82-afb7-46df-9f43-324a55f24992","Type":"ContainerStarted","Data":"ac9fe2e995e0ad2ca3d1c443c96b40060f0d3c091e1409be758484bdc07af02d"} Oct 30 18:40:35.687677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:35.687431 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:35.722240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:35.722188 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" podStartSLOduration=1.365198659 podStartE2EDuration="1.722171579s" totalImagesPullingTime="356.97292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:35.720998598 +0000 UTC m=+2532.545392893" watchObservedRunningTime="2026-10-30 18:40:35.722171579 +0000 UTC m=+2532.546565874" Oct 30 18:40:36.226337 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:40:36.226300 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:40:36.645611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:36.645578 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.557780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:38.557746 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:39.684194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:39.684166 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 18:40:40.454551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:40.454516 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:41.699365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:41.699326 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 18:40:42.265145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:42.265110 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:42.540166 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:42.540033 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:42.640530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:42.640494 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:43.988754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:43.988713 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:44.088016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:44.087981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:44.776784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:44.776745 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:46.781603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:46.781572 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:48.715276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:48.715170 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:51.221533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:51.221477 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:51.322530 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:51.322489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:51.423902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:51.423862 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:53.653621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:53.653587 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:55.879532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:55.879485 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:58.015772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:58.015737 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:59.752313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:40:59.752281 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:01.767767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:01.767730 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:03.596838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:03.596785 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:05.672123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:05.672081 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:05.954304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:05.954201 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:06.226854 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:41:06.226752 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:41:07.174454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:07.174408 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.277458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:07.277420 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.556310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:07.556191 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.668139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:07.668093 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.793625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:07.793587 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.942414 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:07.942379 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:08.049570 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:08.049538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:08.786485 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:08.786453 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:08.885836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:08.885795 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:10.161593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:10.161555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:10.261085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:10.261053 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:12.086908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:12.086878 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:13.851568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:13.851536 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:15.560583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:15.560547 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:16.254597 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:16.254561 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:17.196684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:17.196651 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:18.844381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:18.844346 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:20.740066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:20.740023 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:22.690447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:22.690412 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:24.569554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:24.569520 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b"] Oct 30 18:41:24.570001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:24.569728 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" containerID="cri-o://35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27" gracePeriod=30 Oct 30 18:41:24.626841 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:24.626800 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr"] Oct 30 18:41:24.627118 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:24.627085 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" containerID="cri-o://16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba" gracePeriod=30 Oct 30 18:41:24.658022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:24.657983 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:25.482692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:25.482656 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:26.826855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:26.826779 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:26.927676 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:26.927643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:27.248401 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:27.248362 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:27.503616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:27.503537 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:28.654562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.654525 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd"] Oct 30 18:41:28.660423 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.660403 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.676741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.676713 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd"] Oct 30 18:41:28.681569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.681543 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6babca-serving-cert\"" Oct 30 18:41:28.729728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.729694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:28.765225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.765195 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8956a23d-6d46-4674-8fc2-6d339cc20c6d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.765380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.765241 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-metrics-certs\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.765380 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.765325 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.765473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.765455 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2l9\" (UniqueName: \"kubernetes.io/projected/8956a23d-6d46-4674-8fc2-6d339cc20c6d-kube-api-access-7s2l9\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.816879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.816837 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:28.866474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.866436 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2l9\" (UniqueName: \"kubernetes.io/projected/8956a23d-6d46-4674-8fc2-6d339cc20c6d-kube-api-access-7s2l9\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.866691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.866631 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8956a23d-6d46-4674-8fc2-6d339cc20c6d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.866691 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.866685 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-metrics-certs\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.866872 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.866721 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.866966 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:41:28.866895 4582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-serving-cert: secret "maas-api-e2e-extmod-b-6babca-serving-cert" not found Oct 30 18:41:28.866966 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:41:28.866960 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls podName:8956a23d-6d46-4674-8fc2-6d339cc20c6d nodeName:}" failed. No retries permitted until 2026-10-30 18:41:29.366937802 +0000 UTC m=+2586.191332078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls") pod "maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" (UID: "8956a23d-6d46-4674-8fc2-6d339cc20c6d") : secret "maas-api-e2e-extmod-b-6babca-serving-cert" not found Oct 30 18:41:28.867335 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.867313 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8956a23d-6d46-4674-8fc2-6d339cc20c6d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.869397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.869370 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-metrics-certs\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.884911 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.884852 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2l9\" (UniqueName: \"kubernetes.io/projected/8956a23d-6d46-4674-8fc2-6d339cc20c6d-kube-api-access-7s2l9\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:28.989255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:28.989174 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:29.373280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:29.373242 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:29.375763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:29.375739 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:29.575017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:29.574981 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:29.899357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:29.898850 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd"] Oct 30 18:41:29.928416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:29.928392 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:41:29.985608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:29.985574 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:41:30.066451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.066415 4582 generic.go:356] "Generic (PLEG): container finished" podID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerID="16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba" exitCode=0 Oct 30 18:41:30.066638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.066484 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" Oct 30 18:41:30.066638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.066492 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" event={"ID":"8df61b27-a947-4495-b0ae-0f345bfa9e78","Type":"ContainerDied","Data":"16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba"} Oct 30 18:41:30.066638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.066556 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr" event={"ID":"8df61b27-a947-4495-b0ae-0f345bfa9e78","Type":"ContainerDied","Data":"598ffcb6f3ad6aab1287bea6257ebd423bbb198f0b96c4d2ddceea62fd4c0673"} Oct 30 18:41:30.066638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.066589 4582 scope.go:122] "RemoveContainer" containerID="16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba" Oct 30 18:41:30.068847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.068795 4582 generic.go:356] "Generic (PLEG): container finished" podID="2424245e-41e9-40a7-bc54-73467dbd9949" containerID="35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27" exitCode=0 Oct 30 18:41:30.068977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.068925 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" Oct 30 18:41:30.068977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.068951 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" event={"ID":"2424245e-41e9-40a7-bc54-73467dbd9949","Type":"ContainerDied","Data":"35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27"} Oct 30 18:41:30.069111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.069006 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b" event={"ID":"2424245e-41e9-40a7-bc54-73467dbd9949","Type":"ContainerDied","Data":"71c5d98ddedf6e589a5f4c9efff421244a960a95c45ef8754b5c25e21e8ef69d"} Oct 30 18:41:30.070553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.070529 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" event={"ID":"8956a23d-6d46-4674-8fc2-6d339cc20c6d","Type":"ContainerStarted","Data":"c2bd2e9c49645b05ad04e07617a92f9753d921362809b09c24eb4ae3ecf6a819"} Oct 30 18:41:30.080303 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.080278 4582 scope.go:122] "RemoveContainer" containerID="72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af" Oct 30 18:41:30.083221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083195 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-data\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.083343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083263 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-data\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.083343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083326 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-credential-socket\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.083477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083357 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-certs\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.083477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083415 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-certs\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.083477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083452 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-token\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-token\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.083645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083466 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-data" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.083645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083487 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2424245e-41e9-40a7-bc54-73467dbd9949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2424245e-41e9-40a7-bc54-73467dbd9949-istiod-ca-cert\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.083645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083544 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-data" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.083645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083586 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-socket\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.083645 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083632 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-credential-socket\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083687 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-kube-api-access-2ztx2\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-kube-api-access-2ztx2\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083727 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-envoy\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083872 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-istio-token\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-istio-token\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083922 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-podinfo\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083969 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-envoy\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083869 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-certs" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.083883 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-certs" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.084031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.084001 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-kube-api-access-f2s6j\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-kube-api-access-f2s6j\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.084448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.084037 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2424245e-41e9-40a7-bc54-73467dbd9949-istiod-ca-cert" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:30.084448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.084071 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-socket" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.084448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.084110 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-socket\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.084448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.084167 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df61b27-a947-4495-b0ae-0f345bfa9e78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df61b27-a947-4495-b0ae-0f345bfa9e78-istiod-ca-cert\") pod \"8df61b27-a947-4495-b0ae-0f345bfa9e78\" (UID: \"8df61b27-a947-4495-b0ae-0f345bfa9e78\") " Oct 30 18:41:30.084448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.084229 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2424245e-41e9-40a7-bc54-73467dbd9949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2424245e-41e9-40a7-bc54-73467dbd9949-istio-podinfo\") pod \"2424245e-41e9-40a7-bc54-73467dbd9949\" (UID: \"2424245e-41e9-40a7-bc54-73467dbd9949\") " Oct 30 18:41:30.084686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.084645 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-credential-socket" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.085055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085024 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.085055 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085053 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.085219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085068 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.085219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085082 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.085219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085097 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2424245e-41e9-40a7-bc54-73467dbd9949-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.085219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085111 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.085219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085124 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.085563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085315 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-credential-socket" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.085563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085338 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df61b27-a947-4495-b0ae-0f345bfa9e78-istiod-ca-cert" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:30.085563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.085508 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-socket" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.095737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.095709 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-podinfo" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:30.095737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.095715 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-kube-api-access-2ztx2" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "kube-api-access-2ztx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:30.095926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.095737 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-token" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:30.095926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.095867 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2424245e-41e9-40a7-bc54-73467dbd9949-istio-podinfo" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:30.095926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.095888 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-envoy" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.095926 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.095917 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-istio-token" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:30.096129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.095927 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-kube-api-access-f2s6j" pod "2424245e-41e9-40a7-bc54-73467dbd9949" (UID: "2424245e-41e9-40a7-bc54-73467dbd9949"). InnerVolumeSpecName "kube-api-access-f2s6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:30.097506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.097480 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-envoy" pod "8df61b27-a947-4495-b0ae-0f345bfa9e78" (UID: "8df61b27-a947-4495-b0ae-0f345bfa9e78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.098649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.098619 4582 scope.go:122] "RemoveContainer" containerID="16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba" Oct 30 18:41:30.098983 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:41:30.098962 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba\": container with ID starting with 16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba not found: ID does not exist" containerID="16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba" Oct 30 18:41:30.099069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.098991 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba"} err="failed to get container status \"16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba\": rpc error: code = NotFound desc = could not find container \"16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba\": container with ID starting with 16945591c274f7e63956fd6e5a5c7a6e111403f456ea17b40d5fc195277885ba not found: ID does not exist" Oct 30 18:41:30.099069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.099010 4582 scope.go:122] "RemoveContainer" containerID="72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af" Oct 30 18:41:30.099262 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:41:30.099238 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af\": container with ID starting with 72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af not found: ID does not exist" containerID="72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af" Oct 30 18:41:30.099334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.099272 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af"} err="failed to get container status \"72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af\": rpc error: code = NotFound desc = could not find container \"72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af\": container with ID starting with 72804fe3785d4fb3c167abbb0b63c97fa5d2e482d0bf6bc6fcf5dad8404598af not found: ID does not exist" Oct 30 18:41:30.099334 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.099295 4582 scope.go:122] "RemoveContainer" containerID="35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27" Oct 30 18:41:30.108385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.108369 4582 scope.go:122] "RemoveContainer" containerID="35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27" Oct 30 18:41:30.108660 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:41:30.108642 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27\": container with ID starting with 35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27 not found: ID does not exist" containerID="35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27" Oct 30 18:41:30.108710 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.108666 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27"} err="failed to get container status \"35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27\": rpc error: code = NotFound desc = could not find container \"35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27\": container with ID starting with 35bf829c4562b6beb738b5318cbf77baf562ffb9342b9eda053369b31899bf27 not found: ID does not exist" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186388 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2424245e-41e9-40a7-bc54-73467dbd9949-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186414 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2s6j\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-kube-api-access-f2s6j\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186424 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186437 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df61b27-a947-4495-b0ae-0f345bfa9e78-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186445 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2424245e-41e9-40a7-bc54-73467dbd9949-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186453 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186461 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186469 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztx2\" (UniqueName: \"kubernetes.io/projected/8df61b27-a947-4495-b0ae-0f345bfa9e78-kube-api-access-2ztx2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186478 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186486 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2424245e-41e9-40a7-bc54-73467dbd9949-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.186882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.186495 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df61b27-a947-4495-b0ae-0f345bfa9e78-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.410549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.410518 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr"] Oct 30 18:41:30.414427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.414402 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-bckhr"] Oct 30 18:41:30.446210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.446153 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b"] Oct 30 18:41:30.451306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.451286 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-q2v5b"] Oct 30 18:41:30.677701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:30.677665 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:31.078795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:31.078761 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" event={"ID":"8956a23d-6d46-4674-8fc2-6d339cc20c6d","Type":"ContainerStarted","Data":"a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687"} Oct 30 18:41:31.079385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:31.078814 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:31.114480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:31.114431 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" podStartSLOduration=2.76707839 podStartE2EDuration="3.114412938s" totalImagesPullingTime="347.334548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:31.111542402 +0000 UTC m=+2587.935936693" watchObservedRunningTime="2026-10-30 18:41:31.114412938 +0000 UTC m=+2587.938807247" Oct 30 18:41:31.301683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:31.301610 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" path="/var/lib/kubelet/pods/2424245e-41e9-40a7-bc54-73467dbd9949/volumes" Oct 30 18:41:31.302091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:31.302076 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" path="/var/lib/kubelet/pods/8df61b27-a947-4495-b0ae-0f345bfa9e78/volumes" Oct 30 18:41:31.327351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:31.327308 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:31.430726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:31.430697 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:32.654525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:32.654491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:33.097187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.097115 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:33.197592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.197553 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:33.277699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.277664 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp"] Oct 30 18:41:33.278409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278394 4582 cpu_manager.go:414] "containerMap: removing container" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" Oct 30 18:41:33.278465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278410 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" Oct 30 18:41:33.278465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278426 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" Oct 30 18:41:33.278465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278431 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" Oct 30 18:41:33.278465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278448 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" Oct 30 18:41:33.278465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278453 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" Oct 30 18:41:33.278632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278547 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" Oct 30 18:41:33.278632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.278564 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="2424245e-41e9-40a7-bc54-73467dbd9949" containerName="istio-proxy" Oct 30 18:41:33.283688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.283672 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.304043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.304020 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp"] Oct 30 18:41:33.305602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.305577 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-39c99d-dockercfg-2ljtf\"" Oct 30 18:41:33.348883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.348797 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-39c99d\"" Oct 30 18:41:33.424847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.424780 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b46eba5-ffed-4e96-ad32-dec766983bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.425103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.424971 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97vfx\" (UniqueName: \"kubernetes.io/projected/3b46eba5-ffed-4e96-ad32-dec766983bd2-kube-api-access-97vfx\") pod \"payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.526152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.526113 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97vfx\" (UniqueName: \"kubernetes.io/projected/3b46eba5-ffed-4e96-ad32-dec766983bd2-kube-api-access-97vfx\") pod \"payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.526313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.526272 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b46eba5-ffed-4e96-ad32-dec766983bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.527006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.526983 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b46eba5-ffed-4e96-ad32-dec766983bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.542695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.542664 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97vfx\" (UniqueName: \"kubernetes.io/projected/3b46eba5-ffed-4e96-ad32-dec766983bd2-kube-api-access-97vfx\") pod \"payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.597916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.597892 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:33.896044 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:33.895953 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp"] Oct 30 18:41:33.898845 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:41:33.898803 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b46eba5_ffed_4e96_ad32_dec766983bd2.slice/crio-b08a397e262abc6d588fa7cb82a4114c1779c1dc4d7f0a64fd70844600a49818 WatchSource:0}: Error finding container b08a397e262abc6d588fa7cb82a4114c1779c1dc4d7f0a64fd70844600a49818: Status 404 returned error can't find the container with id b08a397e262abc6d588fa7cb82a4114c1779c1dc4d7f0a64fd70844600a49818 Oct 30 18:41:34.102014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:34.101982 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" event={"ID":"3b46eba5-ffed-4e96-ad32-dec766983bd2","Type":"ContainerStarted","Data":"58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496"} Oct 30 18:41:34.102226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:34.102032 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" event={"ID":"3b46eba5-ffed-4e96-ad32-dec766983bd2","Type":"ContainerStarted","Data":"b08a397e262abc6d588fa7cb82a4114c1779c1dc4d7f0a64fd70844600a49818"} Oct 30 18:41:34.102226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:34.102205 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:34.133972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:34.133917 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" podStartSLOduration=1.13390522 podStartE2EDuration="1.13390522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:34.13385847 +0000 UTC m=+2590.958252759" watchObservedRunningTime="2026-10-30 18:41:34.13390522 +0000 UTC m=+2590.958299511" Oct 30 18:41:34.532512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:34.532480 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:36.226150 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:41:36.226112 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:41:36.283514 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:36.283476 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:36.384124 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:36.384089 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:37.093806 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:37.093773 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 18:41:37.279186 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:37.279150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:37.712313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:37.712280 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:37.980541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:37.980455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:38.187336 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:38.187294 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:38.292146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:38.292037 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:39.819732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:39.819669 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:41.864365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:41.864334 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:42.460505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:42.460454 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:42.574538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:42.574504 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:44.127276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:44.127242 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:44.232533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:44.232497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:44.336445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:44.336415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:44.431808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:44.431737 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:45.111306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:45.111281 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 18:41:46.203687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:46.203651 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:48.085601 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:48.085567 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:49.863698 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:49.863665 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:51.577405 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:51.577372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:51.679902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:51.679873 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:52.818349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:52.818312 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:52.920347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:52.920317 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:53.106192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:53.106098 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:53.368511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:53.368484 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:53.478565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:53.478514 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:53.584150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:53.584116 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:53.688553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:53.688455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:54.166777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:54.166743 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:54.474548 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:54.474465 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:55.548667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:55.548629 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:55.658724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:55.658678 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:57.411126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:57.411095 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:59.217686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:41:59.217654 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:01.161206 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:01.161168 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:02.858036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:02.858001 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:04.687183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:04.687147 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:06.225905 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:42:06.225865 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:42:06.489902 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:06.489610 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:08.118240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:08.118190 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:09.790888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:09.790837 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:11.484022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:11.483977 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:13.234778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:13.234736 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:15.074577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.074544 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf"] Oct 30 18:42:15.075652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.075620 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df61b27-a947-4495-b0ae-0f345bfa9e78" containerName="istio-proxy" Oct 30 18:42:15.079620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.079596 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.103340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.103109 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-39c99d-serving-cert\"" Oct 30 18:42:15.108594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.108568 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf"] Oct 30 18:42:15.120090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.120057 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-metrics-certs\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.120286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.120124 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.120286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.120199 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4053cb6c-e305-435c-bd77-8575aa05d772-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.120417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.120354 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbfl\" (UniqueName: \"kubernetes.io/projected/4053cb6c-e305-435c-bd77-8575aa05d772-kube-api-access-vjbfl\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.147816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.147777 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:15.221631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.221594 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4053cb6c-e305-435c-bd77-8575aa05d772-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.221856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.221739 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbfl\" (UniqueName: \"kubernetes.io/projected/4053cb6c-e305-435c-bd77-8575aa05d772-kube-api-access-vjbfl\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.221929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.221883 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-metrics-certs\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.221976 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.221926 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.222084 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:42:15.222069 4582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-serving-cert: secret "maas-api-e2e-extmod-b-39c99d-serving-cert" not found Oct 30 18:42:15.222147 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:42:15.222133 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls podName:4053cb6c-e305-435c-bd77-8575aa05d772 nodeName:}" failed. No retries permitted until 2026-10-30 18:42:15.722114795 +0000 UTC m=+2632.546509068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls") pod "maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" (UID: "4053cb6c-e305-435c-bd77-8575aa05d772") : secret "maas-api-e2e-extmod-b-39c99d-serving-cert" not found Oct 30 18:42:15.222367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.222346 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4053cb6c-e305-435c-bd77-8575aa05d772-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.224367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.224346 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-metrics-certs\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.238638 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.238609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:15.241508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.241489 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbfl\" (UniqueName: \"kubernetes.io/projected/4053cb6c-e305-435c-bd77-8575aa05d772-kube-api-access-vjbfl\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.394375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.394337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:15.728770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.728673 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.731067 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.731042 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls\") pod \"maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:15.993795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:15.993706 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:16.299395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:16.299315 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf"] Oct 30 18:42:16.303530 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:42:16.303503 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4053cb6c_e305_435c_bd77_8575aa05d772.slice/crio-37399f722e4d4d5e88e4a0669525785303f91e8eaeecdb16c1250b95aa516cc1 WatchSource:0}: Error finding container 37399f722e4d4d5e88e4a0669525785303f91e8eaeecdb16c1250b95aa516cc1: Status 404 returned error can't find the container with id 37399f722e4d4d5e88e4a0669525785303f91e8eaeecdb16c1250b95aa516cc1 Oct 30 18:42:16.379565 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:16.379527 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" event={"ID":"4053cb6c-e305-435c-bd77-8575aa05d772","Type":"ContainerStarted","Data":"37399f722e4d4d5e88e4a0669525785303f91e8eaeecdb16c1250b95aa516cc1"} Oct 30 18:42:16.390563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:16.390525 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:17.252545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:17.252512 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:17.387521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:17.387481 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" event={"ID":"4053cb6c-e305-435c-bd77-8575aa05d772","Type":"ContainerStarted","Data":"82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0"} Oct 30 18:42:17.388056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:17.387613 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:17.427086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:17.426995 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" podStartSLOduration=2.007049314 podStartE2EDuration="2.426981764s" totalImagesPullingTime="419.93245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:17.424363139 +0000 UTC m=+2634.248757500" watchObservedRunningTime="2026-10-30 18:42:17.426981764 +0000 UTC m=+2634.251376083" Oct 30 18:42:18.913730 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:18.913653 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:19.542606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:19.542568 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:20.684146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:20.684115 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:22.358609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:22.358573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:23.401200 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:23.401173 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 18:42:24.032809 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:24.032772 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:25.682008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:25.681966 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:27.436737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.436700 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq"] Oct 30 18:42:27.441705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.441684 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.461897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.461870 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq"] Oct 30 18:42:27.467448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.467418 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6018b4-serving-cert\"" Oct 30 18:42:27.519051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.519018 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:27.578001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.577968 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc61542-a9b2-4137-a526-4aa71dffe281-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.578170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.578012 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-metrics-certs\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.578170 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.578032 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vb6\" (UniqueName: \"kubernetes.io/projected/fdc61542-a9b2-4137-a526-4aa71dffe281-kube-api-access-54vb6\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.578312 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.578292 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.619161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.619133 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:27.679319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.679283 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.679512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.679359 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc61542-a9b2-4137-a526-4aa71dffe281-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.679512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.679399 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-metrics-certs\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.679512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.679425 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vb6\" (UniqueName: \"kubernetes.io/projected/fdc61542-a9b2-4137-a526-4aa71dffe281-kube-api-access-54vb6\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.680207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.680183 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc61542-a9b2-4137-a526-4aa71dffe281-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.681832 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.681793 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-metrics-certs\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.681929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.681910 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.696946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.696888 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vb6\" (UniqueName: \"kubernetes.io/projected/fdc61542-a9b2-4137-a526-4aa71dffe281-kube-api-access-54vb6\") pod \"maas-api-e2e-extmod-b-6018b4-665795b455-28jcq\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:27.721205 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.721153 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:27.755738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:27.755711 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:28.066052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:28.065811 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq"] Oct 30 18:42:28.069288 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:42:28.069259 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc61542_a9b2_4137_a526_4aa71dffe281.slice/crio-69e27a5702a960c0d48de75b306b6213130ae6670b170a6e7c1c61016af3f7ed WatchSource:0}: Error finding container 69e27a5702a960c0d48de75b306b6213130ae6670b170a6e7c1c61016af3f7ed: Status 404 returned error can't find the container with id 69e27a5702a960c0d48de75b306b6213130ae6670b170a6e7c1c61016af3f7ed Oct 30 18:42:28.462199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:28.462169 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" event={"ID":"fdc61542-a9b2-4137-a526-4aa71dffe281","Type":"ContainerStarted","Data":"4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02"} Oct 30 18:42:28.462592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:28.462227 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" event={"ID":"fdc61542-a9b2-4137-a526-4aa71dffe281","Type":"ContainerStarted","Data":"69e27a5702a960c0d48de75b306b6213130ae6670b170a6e7c1c61016af3f7ed"} Oct 30 18:42:28.462592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:28.462298 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:28.504987 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:28.504930 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" podStartSLOduration=1.18623248 podStartE2EDuration="1.504913669s" totalImagesPullingTime="318.681189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:28.503491615 +0000 UTC m=+2645.327885918" watchObservedRunningTime="2026-10-30 18:42:28.504913669 +0000 UTC m=+2645.329307960" Oct 30 18:42:29.396527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:29.396494 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:30.752754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:30.752720 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:30.849038 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:30.849005 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:31.074306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:31.074223 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:31.174062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:31.174028 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:32.950325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:32.950272 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:34.475143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:34.475113 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 18:42:34.647770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:34.647735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:36.225983 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:42:36.225947 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:42:36.402984 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:36.402953 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:38.105996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:38.105963 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:39.731865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:39.731811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:41.343835 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:41.343788 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:41.446740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:41.446704 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:41.693456 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:41.693423 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:42.197234 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:42.197200 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:42.298772 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:42.298734 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:43.515449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:43.515401 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:43.649160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:43.649125 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:45.370726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:45.370679 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:47.062229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:47.062193 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:48.723925 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:48.723890 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:50.465973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:50.465937 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:52.205032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:52.204993 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:53.967127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:53.967091 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:55.952263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:55.952223 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:57.939458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:57.939421 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:59.606019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:42:59.605982 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:00.880266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:00.880227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:00.980429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:00.980398 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:01.827694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:01.827646 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:03.782080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:03.782042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:05.545125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:05.545089 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:06.228043 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:43:06.228008 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:43:07.357390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:07.357337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:08.393816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:08.393764 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:09.032770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:09.032723 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:10.795534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:10.795479 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.461301 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:11.461269 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.560190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:11.560159 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:12.110542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:12.110508 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:12.746475 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:12.746442 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:14.550961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:14.550920 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.437016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:16.436981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.540579 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:16.540541 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.052554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:18.052520 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.140767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:18.140735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.231436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:18.231403 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.370135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:18.370100 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.630623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:19.630576 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.729955 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:19.729915 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.936359 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:19.936280 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.172072 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:20.172029 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.272135 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:20.272061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.430800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:20.430766 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.547551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:20.547468 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:21.208355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:21.208317 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:21.308239 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:21.308202 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:22.164490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:22.164457 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:23.874680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:23.874642 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.272938 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:24.272849 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.371724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:24.371691 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.727905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:24.727861 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.899028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:24.898989 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:25.626014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:25.625975 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:26.146451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:26.146415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:28.073924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:28.073882 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:29.185395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:29.185358 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:29.834762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:29.834721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:31.619499 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:31.619464 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:31.717372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:31.717301 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:33.743874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:33.743811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:34.323532 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:34.323501 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:34.648785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:34.648752 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:34.755376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:34.755342 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:35.247010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:35.246976 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:36.143454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:36.143422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:36.232586 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:43:36.232549 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:43:37.965372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:37.965336 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:39.553001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:39.552969 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:39.691838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:39.691795 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.158034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:41.158005 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.249873 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:41.249802 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.348385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:41.348351 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.762287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:41.762256 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:43.493250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:43.493201 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:45.187260 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:45.187227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:46.860034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:46.860002 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.163815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:47.163780 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.266904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:47.266870 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.373180 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:47.373148 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.602286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:47.602212 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.757838 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:47.757784 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.862542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:47.862465 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:48.019802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:48.019772 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:48.648150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:48.648065 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:48.748139 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:48.748105 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:48.848807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:48.848772 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.086481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:50.086449 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.515515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:50.515464 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.636679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:50.636645 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.733651 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:50.733622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.958800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:50.958759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:51.113140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:51.113107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:51.221577 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:51.221505 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:51.322418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:51.322388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:52.420121 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:52.420070 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:52.519688 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:52.519658 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:53.492779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:53.492743 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:53.593600 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:53.593555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:55.348716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:55.348677 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:57.079056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:57.079023 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:57.200716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:57.200681 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:58.772110 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:43:58.772071 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:00.561790 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:00.561758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:02.203426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:02.203393 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:03.870220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:03.870187 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:05.567403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:05.567366 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:06.229777 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:44:06.229739 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:44:07.305654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:07.305622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:08.971664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:08.971634 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:10.669280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:10.669235 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:12.340188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:12.340155 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:14.088152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:14.088116 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:15.842973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:15.842939 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:16.601501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:16.601415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:17.575541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:17.575504 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:19.272111 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:19.272079 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:20.959801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:20.959767 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:22.862105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:22.862066 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:24.682615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:24.682579 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:25.403092 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:25.403059 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:25.502970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:25.502936 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:27.115879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:27.115845 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:28.938269 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:28.938231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:30.756088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:30.756051 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:32.462126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:32.462094 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:34.190158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:34.190123 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:35.875717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:35.875683 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:36.230302 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:44:36.230214 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:44:37.825695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:37.825658 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:39.716001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:39.715954 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:41.517732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:41.517693 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.293035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:43.292991 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:45.048053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:45.048014 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:46.906043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:46.906011 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:48.638965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:48.638931 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:50.272905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:50.272863 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:52.050240 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:52.050204 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:53.766610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:53.766575 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:55.500105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:55.500070 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:55.592525 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:55.592492 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:55.804687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:55.804599 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:56.225913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:56.225866 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:56.362243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:56.362211 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:56.596869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:56.596772 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:56.859591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:56.859510 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:56.988519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:56.988491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:57.087795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:57.087757 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:57.637910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:57.637874 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:57.736496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:57.736462 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:57.848363 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:57.848333 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:58.957613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:58.957576 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:59.897012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:44:59.896976 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.268361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.268269 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.310869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.310835 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9"] Oct 30 18:45:00.316580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.316557 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.336747 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.336709 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9"] Oct 30 18:45:00.378122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.378091 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:00.404273 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.404237 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfnj\" (UniqueName: \"kubernetes.io/projected/45e89650-fd74-4b46-9008-702c8a426fad-kube-api-access-cjfnj\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.404538 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.404517 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e89650-fd74-4b46-9008-702c8a426fad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.418657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.418630 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.506753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.506716 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e89650-fd74-4b46-9008-702c8a426fad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.506992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.506957 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfnj\" (UniqueName: \"kubernetes.io/projected/45e89650-fd74-4b46-9008-702c8a426fad-kube-api-access-cjfnj\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.507368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.507349 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e89650-fd74-4b46-9008-702c8a426fad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.525727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.525654 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfnj\" (UniqueName: \"kubernetes.io/projected/45e89650-fd74-4b46-9008-702c8a426fad-kube-api-access-cjfnj\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.635167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.635127 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:00.817288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.817206 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.904614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.904578 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.944042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:00.944012 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9"] Oct 30 18:45:00.947527 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:45:00.947501 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e89650_fd74_4b46_9008_702c8a426fad.slice/crio-aadb6c0dc62dfaec8b2f3cb831f8a45764246eeec30f9306eb43011f3338a047 WatchSource:0}: Error finding container aadb6c0dc62dfaec8b2f3cb831f8a45764246eeec30f9306eb43011f3338a047: Status 404 returned error can't find the container with id aadb6c0dc62dfaec8b2f3cb831f8a45764246eeec30f9306eb43011f3338a047 Oct 30 18:45:01.487311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:01.487223 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" event={"ID":"45e89650-fd74-4b46-9008-702c8a426fad","Type":"ContainerStarted","Data":"3a2f3537e341a4fe41a34d11d3981dd63b75612a473ea4eb09ce5eca63aca0a6"} Oct 30 18:45:01.487311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:01.487288 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" event={"ID":"45e89650-fd74-4b46-9008-702c8a426fad","Type":"ContainerStarted","Data":"aadb6c0dc62dfaec8b2f3cb831f8a45764246eeec30f9306eb43011f3338a047"} Oct 30 18:45:01.521712 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:01.521666 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" podStartSLOduration=1.5216536349999998 podStartE2EDuration="1.521653635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:01.520853013 +0000 UTC m=+2798.345247302" watchObservedRunningTime="2026-10-30 18:45:01.521653635 +0000 UTC m=+2798.346047927" Oct 30 18:45:02.284875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:02.284797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:02.495265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:02.495229 4582 generic.go:356] "Generic (PLEG): container finished" podID="45e89650-fd74-4b46-9008-702c8a426fad" containerID="3a2f3537e341a4fe41a34d11d3981dd63b75612a473ea4eb09ce5eca63aca0a6" exitCode=60 Oct 30 18:45:02.495655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:02.495303 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" event={"ID":"45e89650-fd74-4b46-9008-702c8a426fad","Type":"ContainerDied","Data":"3a2f3537e341a4fe41a34d11d3981dd63b75612a473ea4eb09ce5eca63aca0a6"} Oct 30 18:45:02.495655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:02.495610 4582 scope.go:122] "RemoveContainer" containerID="3a2f3537e341a4fe41a34d11d3981dd63b75612a473ea4eb09ce5eca63aca0a6" Oct 30 18:45:03.504052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:03.504017 4582 generic.go:356] "Generic (PLEG): container finished" podID="45e89650-fd74-4b46-9008-702c8a426fad" containerID="3a0e7ba6ff09c8cc2c78746532f498ebafe34c954fc245f47365134419432be0" exitCode=60 Oct 30 18:45:03.504590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:03.504107 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" event={"ID":"45e89650-fd74-4b46-9008-702c8a426fad","Type":"ContainerDied","Data":"3a0e7ba6ff09c8cc2c78746532f498ebafe34c954fc245f47365134419432be0"} Oct 30 18:45:03.504590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:03.504181 4582 scope.go:122] "RemoveContainer" containerID="3a2f3537e341a4fe41a34d11d3981dd63b75612a473ea4eb09ce5eca63aca0a6" Oct 30 18:45:03.504590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:03.504462 4582 scope.go:122] "RemoveContainer" containerID="3a0e7ba6ff09c8cc2c78746532f498ebafe34c954fc245f47365134419432be0" Oct 30 18:45:03.504867 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:45:03.504801 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9_odh-ai-gateway-infra(45e89650-fd74-4b46-9008-702c8a426fad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" podUID="45e89650-fd74-4b46-9008-702c8a426fad" Oct 30 18:45:04.077130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:04.077099 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:04.512856 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:04.512799 4582 scope.go:122] "RemoveContainer" containerID="3a0e7ba6ff09c8cc2c78746532f498ebafe34c954fc245f47365134419432be0" Oct 30 18:45:04.513335 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:45:04.513131 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9_odh-ai-gateway-infra(45e89650-fd74-4b46-9008-702c8a426fad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" podUID="45e89650-fd74-4b46-9008-702c8a426fad" Oct 30 18:45:05.972943 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:05.972904 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:06.228334 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:45:06.228254 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:45:07.758382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:07.758343 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:09.436510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:09.436476 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:11.173799 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:11.173769 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:12.901304 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:12.901264 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:14.659216 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:14.659171 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:15.293316 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:15.293283 4582 scope.go:122] "RemoveContainer" containerID="3a0e7ba6ff09c8cc2c78746532f498ebafe34c954fc245f47365134419432be0" Oct 30 18:45:15.586272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:15.586246 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" event={"ID":"45e89650-fd74-4b46-9008-702c8a426fad","Type":"ContainerStarted","Data":"c7f3b8bedcab78b27e1c9fccef520947c8b5609ad4d8f5dd51b915919f8a7db8"} Oct 30 18:45:16.515674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:16.515637 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:16.595683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:16.595602 4582 generic.go:356] "Generic (PLEG): container finished" podID="45e89650-fd74-4b46-9008-702c8a426fad" containerID="c7f3b8bedcab78b27e1c9fccef520947c8b5609ad4d8f5dd51b915919f8a7db8" exitCode=60 Oct 30 18:45:16.595683 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:16.595671 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" event={"ID":"45e89650-fd74-4b46-9008-702c8a426fad","Type":"ContainerDied","Data":"c7f3b8bedcab78b27e1c9fccef520947c8b5609ad4d8f5dd51b915919f8a7db8"} Oct 30 18:45:16.595967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:16.595727 4582 scope.go:122] "RemoveContainer" containerID="3a0e7ba6ff09c8cc2c78746532f498ebafe34c954fc245f47365134419432be0" Oct 30 18:45:16.596161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:16.596144 4582 scope.go:122] "RemoveContainer" containerID="c7f3b8bedcab78b27e1c9fccef520947c8b5609ad4d8f5dd51b915919f8a7db8" Oct 30 18:45:16.596513 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:45:16.596491 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9_odh-ai-gateway-infra(45e89650-fd74-4b46-9008-702c8a426fad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" podUID="45e89650-fd74-4b46-9008-702c8a426fad" Oct 30 18:45:16.634183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:16.634160 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9"] Oct 30 18:45:16.735880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:16.735842 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:17.781382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:17.781352 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:17.838707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:17.838678 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e89650-fd74-4b46-9008-702c8a426fad-kube-api-access-cjfnj\" (UniqueName: \"kubernetes.io/projected/45e89650-fd74-4b46-9008-702c8a426fad-kube-api-access-cjfnj\") pod \"45e89650-fd74-4b46-9008-702c8a426fad\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " Oct 30 18:45:17.838947 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:17.838806 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e89650-fd74-4b46-9008-702c8a426fad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e89650-fd74-4b46-9008-702c8a426fad-maas-api-service-ca\") pod \"45e89650-fd74-4b46-9008-702c8a426fad\" (UID: \"45e89650-fd74-4b46-9008-702c8a426fad\") " Oct 30 18:45:17.839172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:17.839150 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e89650-fd74-4b46-9008-702c8a426fad-maas-api-service-ca" pod "45e89650-fd74-4b46-9008-702c8a426fad" (UID: "45e89650-fd74-4b46-9008-702c8a426fad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:17.840940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:17.840908 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e89650-fd74-4b46-9008-702c8a426fad-kube-api-access-cjfnj" pod "45e89650-fd74-4b46-9008-702c8a426fad" (UID: "45e89650-fd74-4b46-9008-702c8a426fad"). InnerVolumeSpecName "kube-api-access-cjfnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:17.940515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:17.940483 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e89650-fd74-4b46-9008-702c8a426fad-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:45:17.940515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:17.940510 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjfnj\" (UniqueName: \"kubernetes.io/projected/45e89650-fd74-4b46-9008-702c8a426fad-kube-api-access-cjfnj\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:45:18.611520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:18.611441 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" Oct 30 18:45:18.611520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:18.611455 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9" event={"ID":"45e89650-fd74-4b46-9008-702c8a426fad","Type":"ContainerDied","Data":"aadb6c0dc62dfaec8b2f3cb831f8a45764246eeec30f9306eb43011f3338a047"} Oct 30 18:45:18.611759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:18.611525 4582 scope.go:122] "RemoveContainer" containerID="c7f3b8bedcab78b27e1c9fccef520947c8b5609ad4d8f5dd51b915919f8a7db8" Oct 30 18:45:18.659434 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:18.659409 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9"] Oct 30 18:45:18.671701 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:18.671677 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889765-w75l9"] Oct 30 18:45:18.892941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:18.892899 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:19.302327 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:19.302256 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e89650-fd74-4b46-9008-702c8a426fad" path="/var/lib/kubelet/pods/45e89650-fd74-4b46-9008-702c8a426fad/volumes" Oct 30 18:45:20.858333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:20.858289 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:20.967864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:20.967831 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:21.058020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:21.057990 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:21.763893 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:21.763858 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:21.865198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:21.865142 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:23.751418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:23.751378 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:23.850588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:23.850551 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:23.953991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:23.953958 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:24.049537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:24.049438 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:25.911212 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:25.911178 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:27.751857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:27.751808 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:29.536124 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:29.536087 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:31.073966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:31.073922 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:31.176740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:31.176712 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:31.449598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:31.449566 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:33.457652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:33.457595 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:35.235089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:35.235058 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:35.906387 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:35.906349 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:36.013673 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:36.013640 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:36.225883 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:45:36.225773 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:45:36.381551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:36.381502 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:37.570182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:37.570138 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:38.075395 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:38.075361 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:38.225655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:38.225603 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:38.935326 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:38.935284 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:39.870608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:39.870573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:41.756082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:41.756042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:41.969318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:41.969285 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:42.069695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:42.069609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:42.557836 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:42.557774 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:42.669101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:42.669031 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:42.784411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:42.784375 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:43.804956 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:43.804924 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:43.937923 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:43.937884 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:44.055636 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:44.055546 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:46.079222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:46.079186 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:47.798602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:47.798570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:48.869052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:48.869015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:48.995547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:48.995512 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.671392 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:49.671355 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.832795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:49.832759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.943089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:49.943005 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:51.208759 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:51.208728 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:52.164656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:52.164621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:53.889074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:53.889038 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:55.163668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:55.163618 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:55.645670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:55.645638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:55.752709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:55.752676 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:55.961614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:55.961510 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:58.335542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:45:58.335499 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:00.006622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:00.006590 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.848367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:01.848339 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:02.155476 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:02.155443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:02.264346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:02.264314 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:03.976569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:03.976542 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:05.770004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:05.769975 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:06.226141 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:46:06.226090 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:46:07.579352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:07.579319 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:09.362205 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:09.362169 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:11.055448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:11.055406 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:12.790144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:12.790110 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:14.283190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:14.283161 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:14.383401 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:14.383366 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:14.833721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:14.833675 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:15.013917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:15.013883 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:16.689589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:16.689553 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:16.905116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:16.905076 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:17.830643 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:17.830601 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:17.929266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:17.929234 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:18.910424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:18.910375 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:20.600179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:20.600148 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:22.325849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:22.325798 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:24.194441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:24.194384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:24.858858 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:24.858798 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:25.983210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:25.983173 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:27.768765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:27.768724 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:28.455631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:28.455575 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:29.587743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:29.587689 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:31.621542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:31.621506 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:31.717545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:31.717504 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:31.924794 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:31.924707 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:32.033488 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:32.033432 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:33.867325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:33.867287 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:34.500928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:34.500868 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:34.611959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:34.611917 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:34.712892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:34.712851 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:35.666775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:35.666737 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:36.226906 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:46:36.226864 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:46:37.482465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:37.482421 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:37.708740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:37.708697 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:37.810155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:37.810082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:38.043739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:38.043704 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:38.247411 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:38.247379 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:38.442016 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:38.441980 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:38.566341 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:38.566249 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:38.722931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:38.722898 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:39.409958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:39.409925 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:39.508147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:39.508115 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:40.097146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:40.097105 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:40.339860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:40.339809 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:40.452879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:40.452814 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:40.767507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:40.767431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:41.305390 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:41.305338 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:41.418181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:41.418147 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:42.760763 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:42.760730 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:44.553880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:44.553845 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:46.218377 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:46.218336 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:48.055861 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:48.055799 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:49.711071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:49.710977 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:49.804589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:49.804550 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:51.565949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:51.565910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:51.846975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:51.846893 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:52.083209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:52.083175 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:52.485428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:52.485364 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:52.585667 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:52.585635 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:52.771429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:52.771348 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:53.719474 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:53.719440 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:54.676834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:54.676793 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:55.451369 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.451328 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm"] Oct 30 18:46:55.452130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.452111 4582 cpu_manager.go:414] "containerMap: removing container" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:46:55.452130 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.452131 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:46:55.452246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.452153 4582 cpu_manager.go:414] "containerMap: removing container" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:46:55.452246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.452159 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:46:55.452313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.452275 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:46:55.452313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.452289 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:46:55.455991 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.455970 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.476169 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.476144 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm"] Oct 30 18:46:55.559962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.559922 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xfzjq\"" Oct 30 18:46:55.583562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583533 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.583732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583577 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.583732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583648 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvssk\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-kube-api-access-fvssk\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.583863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583745 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.583863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583775 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.583962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583913 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.583962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583938 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.584084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.583990 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5030f1-3eae-4d64-9a64-52ee30032821-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.584148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.584105 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685290 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685326 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685357 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvssk\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-kube-api-access-fvssk\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685386 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685403 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685456 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685485 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685545 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5030f1-3eae-4d64-9a64-52ee30032821-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685618 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685704 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685807 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.685942 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.685886 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.686214 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.686025 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.686339 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.686321 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5030f1-3eae-4d64-9a64-52ee30032821-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.687882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.687862 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.688056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.688037 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.701635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.701571 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.702050 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.702031 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvssk\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-kube-api-access-fvssk\") pod \"maas-default-gateway-openshift-default-76597d6855-5hmsm\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:55.771700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:55.771678 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:56.090675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.090651 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm"] Oct 30 18:46:56.093571 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:46:56.093544 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5030f1_3eae_4d64_9a64_52ee30032821.slice/crio-1e79e2d4f5ecfa33f02267a170d8ba6f5d3209e2cee020cc3fa0f885758fbe0e WatchSource:0}: Error finding container 1e79e2d4f5ecfa33f02267a170d8ba6f5d3209e2cee020cc3fa0f885758fbe0e: Status 404 returned error can't find the container with id 1e79e2d4f5ecfa33f02267a170d8ba6f5d3209e2cee020cc3fa0f885758fbe0e Oct 30 18:46:56.096331 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.096299 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:46:56.096416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.096358 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:46:56.096416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.096389 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:46:56.290874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.290809 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" event={"ID":"9a5030f1-3eae-4d64-9a64-52ee30032821","Type":"ContainerStarted","Data":"2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea"} Oct 30 18:46:56.291053 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.290947 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" event={"ID":"9a5030f1-3eae-4d64-9a64-52ee30032821","Type":"ContainerStarted","Data":"1e79e2d4f5ecfa33f02267a170d8ba6f5d3209e2cee020cc3fa0f885758fbe0e"} Oct 30 18:46:56.328255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.328202 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podStartSLOduration=1.328189248 podStartE2EDuration="1.328189248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:56.328076375 +0000 UTC m=+2913.152470666" watchObservedRunningTime="2026-10-30 18:46:56.328189248 +0000 UTC m=+2913.152583597" Oct 30 18:46:56.349093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.349021 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:56.772241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:56.772215 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:46:57.774503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:57.774470 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:46:57.775028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:57.774519 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 18:46:58.176156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:58.176126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:58.772980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:58.772941 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:46:58.773172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:58.773043 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 18:46:59.772024 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:59.771991 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:46:59.772541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:59.772040 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 18:46:59.939974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:46:59.939931 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:00.039627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:00.039544 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:00.140512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:00.140476 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:00.241015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:00.240984 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:00.368121 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:00.368090 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:00.772738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:00.772644 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:47:00.772738 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:00.772704 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 18:47:01.274515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:01.274477 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:01.369168 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:01.369131 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:01.774179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:01.774144 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:47:01.774757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:01.774722 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 18:47:01.868142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:01.868111 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:02.189573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:02.189538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:02.633734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:02.633692 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:02.772105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:02.772075 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:47:02.772288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:02.772139 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 18:47:03.772664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:03.772628 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:47:03.773095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:03.772710 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 18:47:04.464866 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:04.464797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:04.997978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:04.997941 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:47:04.998502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:04.998380 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:47:04.999119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:04.999100 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 18:47:06.228213 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:47:06.228173 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:47:06.253023 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:06.252991 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:08.389338 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:08.389305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:08.490314 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:08.490273 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:08.621536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:08.621502 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:08.718610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:08.718520 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:10.765765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:10.765695 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:12.645589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:12.645543 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:14.380508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:14.380475 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:14.594157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:14.594107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:14.697122 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:14.697027 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:14.906596 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:14.906558 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:15.007740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:15.007652 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:15.444372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:15.444337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:16.062936 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:16.062905 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:16.385783 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:16.385728 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:16.982210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:16.982171 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:18.085884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:18.085843 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:18.835204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:18.835167 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:18.936346 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:18.936306 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:20.081207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:20.081176 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:20.740572 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:20.740533 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:20.840236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:20.840201 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:21.391221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:21.391180 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:22.396959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:22.396925 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:24.271432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:24.271391 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:26.029653 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:26.029621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:27.177699 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:27.177667 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:27.276857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:27.276798 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:28.578885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:28.578853 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:28.681868 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:28.681803 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:29.307913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:29.307884 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:29.408089 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:29.408053 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:29.508146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:29.508107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:29.993315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:29.993282 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:30.970615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:30.970582 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:32.853268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:32.853226 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:34.650674 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:34.650634 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:35.282867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:35.282832 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:35.388520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:35.388484 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:36.226064 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:47:36.226030 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:47:36.759543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:36.759506 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:37.799708 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:37.799657 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:37.910810 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:37.910770 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:38.994795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:38.994758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:40.773888 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:40.773847 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:42.565739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:42.565698 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:42.855147 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:42.855026 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:42.962054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:42.962021 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:43.646479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:43.646445 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:44.518937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:44.518895 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:44.728442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:44.728412 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:46.093543 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:46.093497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:46.190953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:46.190878 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:46.700793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:46.700754 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:48.184881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:48.184849 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:49.278569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:49.278538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:49.853625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:49.853592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:51.274680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:51.274641 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:51.373544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:51.373481 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:51.472100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:51.472062 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:54.069492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:54.069451 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:55.816158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:55.816126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:57.575330 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:57.575293 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:59.122547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:59.122497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:59.212085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:59.212048 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:59.372786 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:47:59.372699 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:01.449165 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:01.449136 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:03.047569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:03.047537 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:03.645604 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:03.645562 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:05.788203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:05.788167 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:06.226221 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:48:06.226184 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:48:07.568481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:07.568441 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:08.956752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:08.956719 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:09.063555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:09.063521 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:09.361593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:09.361515 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:10.656119 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.656080 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk"] Oct 30 18:48:10.657196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.657175 4582 cpu_manager.go:414] "containerMap: removing container" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:48:10.657277 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.657198 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:48:10.657464 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.657449 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e89650-fd74-4b46-9008-702c8a426fad" containerName="cleanup" Oct 30 18:48:10.661503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.661485 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.692830 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.692780 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk"] Oct 30 18:48:10.781781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.781747 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.781781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.781782 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.782082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.781804 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.782082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.781929 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.782082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.781981 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.782082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.782045 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.782223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.782116 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.782223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.782161 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndms\" (UniqueName: \"kubernetes.io/projected/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-kube-api-access-qndms\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.782294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.782231 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.883626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883595 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.883816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883636 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndms\" (UniqueName: \"kubernetes.io/projected/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-kube-api-access-qndms\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.883816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883679 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.883816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883738 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.883816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883755 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.883816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883783 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.883816 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883847 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.884235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.883971 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.884235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.884088 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.884235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.884128 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.884516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.884490 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.884647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.884524 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.884647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.884618 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.884785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.884641 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.886188 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.886169 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.886523 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.886502 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.900082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.900060 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.900452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.900431 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndms\" (UniqueName: \"kubernetes.io/projected/cd66eba8-79c3-4cb5-8b81-ac65cac3619d-kube-api-access-qndms\") pod \"maas-default-gateway-openshift-default-76597d6855-5cfxk\" (UID: \"cd66eba8-79c3-4cb5-8b81-ac65cac3619d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:10.976020 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:10.975913 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:11.186677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.186635 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:11.295953 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:48:11.295919 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd66eba8_79c3_4cb5_8b81_ac65cac3619d.slice/crio-27405d16fd2f3d7ad3f1a00340fe39657326cf32164d87b2d6fb80e5bfaeacd9 WatchSource:0}: Error finding container 27405d16fd2f3d7ad3f1a00340fe39657326cf32164d87b2d6fb80e5bfaeacd9: Status 404 returned error can't find the container with id 27405d16fd2f3d7ad3f1a00340fe39657326cf32164d87b2d6fb80e5bfaeacd9 Oct 30 18:48:11.298946 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.298910 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:48:11.299066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.298978 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:48:11.299066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.299006 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:48:11.303043 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.303022 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk"] Oct 30 18:48:11.823608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.823561 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" event={"ID":"cd66eba8-79c3-4cb5-8b81-ac65cac3619d","Type":"ContainerStarted","Data":"a4b9e28487bfc5168a1c085f1ed4efaeebe16c522ad5f8d08536404d86c82746"} Oct 30 18:48:11.824077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.823628 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" event={"ID":"cd66eba8-79c3-4cb5-8b81-ac65cac3619d","Type":"ContainerStarted","Data":"27405d16fd2f3d7ad3f1a00340fe39657326cf32164d87b2d6fb80e5bfaeacd9"} Oct 30 18:48:11.874162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.874106 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podStartSLOduration=1.874087576 podStartE2EDuration="1.874087576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:11.869738354 +0000 UTC m=+2988.694132654" watchObservedRunningTime="2026-10-30 18:48:11.874087576 +0000 UTC m=+2988.698481870" Oct 30 18:48:11.976318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:11.976284 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:12.963845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:12.960352 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:12.978511 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:12.978479 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 18:48:12.978675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:12.978531 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 18:48:13.976724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:13.976682 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 18:48:13.977161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:13.976751 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 18:48:14.665844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:14.663409 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:14.977094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:14.977011 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 18:48:14.977094 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:14.977081 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 18:48:15.977093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:15.977055 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 18:48:15.977503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:15.977122 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 18:48:16.475372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:16.475336 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:16.976236 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:16.976200 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 18:48:16.976432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:16.976252 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 18:48:17.163846 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:17.149124 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:17.976296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:17.976261 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 18:48:17.976473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:17.976321 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 18:48:18.133106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:18.133064 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.233379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:18.233278 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.336178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:18.336126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.976922 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:18.976885 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 18:48:18.977112 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:18.976944 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 18:48:19.640142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:19.640103 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:19.974100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:19.974022 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:19.981512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:19.981484 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:19.981928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:19.981915 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:19.982552 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:19.982534 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" Oct 30 18:48:21.777578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:21.777549 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:23.467440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:23.467404 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.132104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.132070 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.719583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.719547 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s"] Oct 30 18:48:25.725385 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.725362 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.744850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.744802 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s"] Oct 30 18:48:25.795265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795229 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvpm\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-kube-api-access-6wvpm\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795342 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795420 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795445 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795568 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795671 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795701 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795796 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795780 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.795898 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.795871 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.857080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.857039 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.897679 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.897646 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.897871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.897747 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvpm\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-kube-api-access-6wvpm\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.897871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.897812 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.897972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.897933 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.897972 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898014 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898009 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898106 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898141 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898132 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898243 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898183 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898303 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898375 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898450 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898729 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898704 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.898855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.898832 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.900473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.900449 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.900767 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.900750 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.915274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.915248 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvpm\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-kube-api-access-6wvpm\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.915496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.915473 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-p6n4s\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:25.972427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:25.972359 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:26.057908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.057876 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:26.374244 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.374212 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s"] Oct 30 18:48:26.377508 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:48:26.377481 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2e7559_dbb4_4e25_b7d5_4acb80518a7f.slice/crio-6cc87de0bf56a25f1d11ef8edb55d22060dd73a6b1d7f1a5a62ad5340440dae0 WatchSource:0}: Error finding container 6cc87de0bf56a25f1d11ef8edb55d22060dd73a6b1d7f1a5a62ad5340440dae0: Status 404 returned error can't find the container with id 6cc87de0bf56a25f1d11ef8edb55d22060dd73a6b1d7f1a5a62ad5340440dae0 Oct 30 18:48:26.380235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.380203 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:48:26.380318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.380268 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:48:26.380318 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.380296 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:48:26.954717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.954678 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" event={"ID":"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f","Type":"ContainerStarted","Data":"cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581"} Oct 30 18:48:26.954965 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.954752 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" event={"ID":"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f","Type":"ContainerStarted","Data":"6cc87de0bf56a25f1d11ef8edb55d22060dd73a6b1d7f1a5a62ad5340440dae0"} Oct 30 18:48:26.994954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.994906 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podStartSLOduration=1.9948921020000001 podStartE2EDuration="1.994892102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:26.98916636 +0000 UTC m=+3003.813560652" watchObservedRunningTime="2026-10-30 18:48:26.994892102 +0000 UTC m=+3003.819286393" Oct 30 18:48:26.999471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:26.999447 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:27.058889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:27.058853 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:27.060874 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:27.060842 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:27.061009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:27.060890 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:28.028447 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:28.028411 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:28.061291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:28.061248 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:28.061585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:28.061555 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:28.719848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:28.719776 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:29.067265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:29.067181 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:29.067642 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:29.067259 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:29.164845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:29.150254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:29.541491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:29.541460 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:30.061847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:30.059187 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:30.061847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:30.059250 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:30.845204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:30.845164 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:31.062237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:31.062199 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:31.062527 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:31.062499 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:32.067264 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:32.067229 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:32.067697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:32.067284 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:32.655800 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:32.655759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:33.059070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:33.058977 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:33.059070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:33.059033 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:34.058883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:34.058800 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 18:48:34.059305 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:34.058913 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 18:48:34.444224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:34.444156 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:35.063202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:35.063173 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:35.063675 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:35.063656 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:35.064325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:35.064305 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 18:48:36.225944 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:48:36.225907 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:48:36.361832 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:36.361779 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:36.763374 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:36.763337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:36.883994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:36.883963 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:37.008671 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:37.008632 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:38.598104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:38.598070 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:38.807077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:38.807043 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:38.924616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:38.924584 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:40.614313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:40.614279 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:42.364618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:42.364589 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:44.158668 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:44.158632 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:45.956990 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:45.956956 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:47.757192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:47.757155 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:49.503213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:49.503175 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:51.274348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:51.274316 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:53.070224 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:53.070189 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:54.747195 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:54.747160 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:56.477534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:56.477502 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:58.154480 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:58.154450 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:59.944608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:48:59.944572 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:01.741439 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:01.741408 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:03.476307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:03.476269 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:05.264529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:05.264494 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:06.226917 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:49:06.226883 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:49:06.989658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:06.989620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:07.242665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:07.242586 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:07.861883 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:07.861850 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:08.751190 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:08.751137 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:10.604994 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:10.604955 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:12.595497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:12.595456 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:14.284075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:14.284045 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.164386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:16.164333 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.240944 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:17.240912 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.426709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:17.426669 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.945175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:17.945137 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:19.683226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:19.683191 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:21.482581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:21.482549 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.234255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.234220 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h"] Oct 30 18:49:23.238870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.238853 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.257490 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.257458 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h"] Oct 30 18:49:23.261760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.261742 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0cc6b1-serving-cert\"" Oct 30 18:49:23.304350 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.304315 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.352481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.352449 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws4g2\" (UniqueName: \"kubernetes.io/projected/0cb1fe32-df10-442d-917f-090ee73cfb32-kube-api-access-ws4g2\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.352657 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.352601 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb1fe32-df10-442d-917f-090ee73cfb32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.352731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.352718 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-maas-api-tls\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.352869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.352856 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-metrics-certs\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.390347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.390312 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.453954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.453924 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb1fe32-df10-442d-917f-090ee73cfb32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.454127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.453999 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-maas-api-tls\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.454127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.454067 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-metrics-certs\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.454262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.454192 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws4g2\" (UniqueName: \"kubernetes.io/projected/0cb1fe32-df10-442d-917f-090ee73cfb32-kube-api-access-ws4g2\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.454555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.454528 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb1fe32-df10-442d-917f-090ee73cfb32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.456424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.456407 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-metrics-certs\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.465320 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.465301 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0cc6b1-serving-cert\"" Oct 30 18:49:23.471261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.471237 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws4g2\" (UniqueName: \"kubernetes.io/projected/0cb1fe32-df10-442d-917f-090ee73cfb32-kube-api-access-ws4g2\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.477253 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.477234 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-maas-api-tls\") pod \"maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.492069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.491983 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.554400 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.554364 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:23.859440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.859414 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h"] Oct 30 18:49:23.866222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:23.866196 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:49:24.351808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:24.351775 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" event={"ID":"0cb1fe32-df10-442d-917f-090ee73cfb32","Type":"ContainerStarted","Data":"6752affe7d655463946bd1aa312bfd780f3cb7e6e88391a4385f6b6bbfb83b60"} Oct 30 18:49:24.352194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:24.352081 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:24.352194 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:24.352130 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" event={"ID":"0cb1fe32-df10-442d-917f-090ee73cfb32","Type":"ContainerStarted","Data":"71104002ddc02fa413bdd7ed5d91a827975fd5d2a3d5145b38ffbd82f473b3d6"} Oct 30 18:49:24.388268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:24.388222 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" podStartSLOduration=0.970002712 podStartE2EDuration="1.388209116s" totalImagesPullingTime="418.206404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:24.384760674 +0000 UTC m=+3061.209154966" watchObservedRunningTime="2026-10-30 18:49:24.388209116 +0000 UTC m=+3061.212603428" Oct 30 18:49:25.103504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:25.103473 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:25.260088 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:25.260052 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:25.701465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:25.701432 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:26.066080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:26.065989 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:26.172554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:26.172518 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:26.826143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:26.826111 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:27.259162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:27.259126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:27.366142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:27.366107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:27.460031 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:27.459991 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:28.076576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:28.076531 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:29.265426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:29.265398 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:30.366368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:30.366340 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:49:31.028793 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:31.028749 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:32.898719 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:32.898689 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:34.150103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:34.150068 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:34.270774 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:34.270730 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:34.459449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:34.459368 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:34.871855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:34.871796 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:35.311116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:35.311019 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:35.411666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:35.411633 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:35.648666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:35.648634 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:36.226114 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:49:36.226078 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:49:37.036837 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:37.036787 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:37.135757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:37.135726 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:37.819775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:37.819739 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:38.336782 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:38.336753 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:38.587379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:38.587281 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:38.681167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:38.681126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:38.874661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:38.874621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:40.413786 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:40.413753 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:40.760899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:40.759481 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:40.892477 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:40.892429 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:42.183193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:42.183163 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:42.312726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:42.312697 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:42.657472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:42.657436 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:42.935131 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:42.935027 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:44.283424 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:44.283384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:46.186592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:46.186560 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:46.809929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:46.809884 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:46.926954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:46.926906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:48.063133 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:48.063101 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:48.225221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:48.225190 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:48.326897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:48.326771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:48.546631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:48.546544 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:48.643563 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:48.643524 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:49.969202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:49.969164 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:51.743368 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:51.743328 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:51.844620 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:51.844591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:51.945282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:51.945246 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:53.133046 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:53.133011 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:54.170739 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:54.170707 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:55.961467 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:55.961435 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:57.708972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:57.708930 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:59.659501 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:49:59.659457 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:01.364731 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:01.364696 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:03.072245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:03.072212 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:04.762469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:04.762435 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:06.226978 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:50:06.226936 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:50:06.423309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:06.423269 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:08.176071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:08.176038 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:09.860892 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:09.860856 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:11.664733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:11.664691 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:13.489807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:13.489770 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:15.260989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:15.260952 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:16.100397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:16.100360 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:16.765262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:16.765220 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:17.296967 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:17.296929 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:17.401101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:17.401061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:19.313376 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:19.313332 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:21.085517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:21.085480 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:23.045129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:23.045093 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:24.845417 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:24.845380 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:26.591798 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:26.591682 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:27.580505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:27.580474 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:27.849149 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:27.849067 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:28.373203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:28.373171 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:28.484578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:28.484530 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:28.829106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:28.829008 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:28.992749 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:28.992697 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:30.467723 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:30.467688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:32.563711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:32.563672 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:32.996246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:32.996206 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:33.105291 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:33.105254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:34.815887 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:34.805737 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:34.938505 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:34.938458 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:35.094792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:35.094701 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:36.062007 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:36.061969 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5cfxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:36.063103 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:36.063073 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5cfxk" podUID="cd66eba8-79c3-4cb5-8b81-ac65cac3619d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.063610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:36.063587 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-5hmsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:36.063802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:36.063779 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.136980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:36.136946 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-p6n4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:36.137184 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:36.137023 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.295775 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:50:36.295732 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:50:37.538555 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.538518 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw"] Oct 30 18:50:37.542229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.542206 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.563869 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.563836 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1aa59a-serving-cert\"" Oct 30 18:50:37.568057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.568031 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw"] Oct 30 18:50:37.604945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.604910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:37.651734 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.651703 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.651928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.651850 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.651928 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.651888 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dspdd\" (UniqueName: \"kubernetes.io/projected/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-kube-api-access-dspdd\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.652042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.652015 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-metrics-certs\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.690315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.690284 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:37.753595 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.753554 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.753803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.753609 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dspdd\" (UniqueName: \"kubernetes.io/projected/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-kube-api-access-dspdd\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.753803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.753718 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-metrics-certs\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.753970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.753848 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.754381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.754357 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.757372 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.757334 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-metrics-certs\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.757743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.757722 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.771566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.771529 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dspdd\" (UniqueName: \"kubernetes.io/projected/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-kube-api-access-dspdd\") pod \"maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:37.812157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.812078 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:37.855167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:37.855122 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:38.166517 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:38.166495 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw"] Oct 30 18:50:38.168969 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:50:38.168941 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24a3e46_8357_43b8_ab2c_9f41aaa929a4.slice/crio-1288ea798d05347e721582146bbe818cb86f62f08baac1282940547efb116b68 WatchSource:0}: Error finding container 1288ea798d05347e721582146bbe818cb86f62f08baac1282940547efb116b68: Status 404 returned error can't find the container with id 1288ea798d05347e721582146bbe818cb86f62f08baac1282940547efb116b68 Oct 30 18:50:39.027686 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:39.027653 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" event={"ID":"f24a3e46-8357-43b8-ab2c-9f41aaa929a4","Type":"ContainerStarted","Data":"2a76951b264549d22584dc208b6bf64066ca5ac3dd7ee7299cc410a2a19f4de4"} Oct 30 18:50:39.028220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:39.027712 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" event={"ID":"f24a3e46-8357-43b8-ab2c-9f41aaa929a4","Type":"ContainerStarted","Data":"1288ea798d05347e721582146bbe818cb86f62f08baac1282940547efb116b68"} Oct 30 18:50:39.028220 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:39.027851 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:39.071293 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:39.071245 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" podStartSLOduration=1.6972951520000001 podStartE2EDuration="2.0712316s" totalImagesPullingTime="373.936448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:39.066940306 +0000 UTC m=+3135.891334599" watchObservedRunningTime="2026-10-30 18:50:39.0712316 +0000 UTC m=+3135.895625892" Oct 30 18:50:39.552287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:39.552255 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:41.361193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:41.361148 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:43.151074 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:43.151039 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:44.845176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:44.845147 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.039507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:45.039472 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 18:50:46.525504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:46.525470 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:48.248105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:48.248066 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:49.954225 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:49.954184 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:50.843075 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:50.843034 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:50.941171 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:50.941140 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:52.065018 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:52.064973 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:53.860082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:53.860044 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:55.649001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:55.648966 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:57.472900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:57.472867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:59.558802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:59.558760 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:59.665640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:50:59.665608 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.222603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:01.222569 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:02.969875 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:02.969840 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:03.928545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:03.928509 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:04.030466 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:04.030433 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:04.970954 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:04.970920 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:05.870093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:05.870062 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:05.969266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:05.969230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:06.257891 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:51:06.257770 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:51:06.974238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:06.974182 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:08.943090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:08.943054 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:10.772610 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:10.772573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:12.720238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:12.720200 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:14.426724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:14.426688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:16.146261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:16.146230 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:17.451632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:17.451590 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:17.946700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:17.946646 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:19.632755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:19.632670 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:21.371728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:21.371692 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:23.086070 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:23.086035 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:24.663229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.663195 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr"] Oct 30 18:51:24.668057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.668028 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.689491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.689467 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c52ec7-serving-cert\"" Oct 30 18:51:24.695052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.695025 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr"] Oct 30 18:51:24.732603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.732573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:24.770102 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.770069 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.770352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.770329 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-maas-api-tls\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.770452 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.770434 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-metrics-certs\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.770520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.770480 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625wc\" (UniqueName: \"kubernetes.io/projected/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-kube-api-access-625wc\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.812752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.812724 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:24.872270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.872240 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.872460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.872412 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-maas-api-tls\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.872460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.872456 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-metrics-certs\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.872571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.872482 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625wc\" (UniqueName: \"kubernetes.io/projected/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-kube-api-access-625wc\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.873056 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.873024 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.875786 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.875758 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-maas-api-tls\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.876544 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.876522 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-metrics-certs\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.888664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.888638 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625wc\" (UniqueName: \"kubernetes.io/projected/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-kube-api-access-625wc\") pod \"maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:24.930144 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.930072 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:24.982989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:24.982958 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:25.281654 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:25.281626 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr"] Oct 30 18:51:25.284224 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:51:25.284197 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd7a1dd_60de_4624_9d5a_5e36f75aedb1.slice/crio-831b724b61620f349a46528693ce7fed468685541f68f2673f700cfd8b6de38d WatchSource:0}: Error finding container 831b724b61620f349a46528693ce7fed468685541f68f2673f700cfd8b6de38d: Status 404 returned error can't find the container with id 831b724b61620f349a46528693ce7fed468685541f68f2673f700cfd8b6de38d Oct 30 18:51:25.389801 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:25.389762 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" event={"ID":"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1","Type":"ContainerStarted","Data":"831b724b61620f349a46528693ce7fed468685541f68f2673f700cfd8b6de38d"} Oct 30 18:51:26.398560 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:26.398520 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" event={"ID":"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1","Type":"ContainerStarted","Data":"281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f"} Oct 30 18:51:26.399286 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:26.399259 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:26.434063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:26.434006 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" podStartSLOduration=2.001381201 podStartE2EDuration="2.433992302s" totalImagesPullingTime="432.611101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:26.432204372 +0000 UTC m=+3183.256598665" watchObservedRunningTime="2026-10-30 18:51:26.433992302 +0000 UTC m=+3183.258386594" Oct 30 18:51:26.655410 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:26.655324 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:28.478586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:28.478553 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:30.596404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:30.596368 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:32.441857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:32.439036 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:51:33.343129 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.343070 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:33.590564 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.590531 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:33.710142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.710107 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:33.911230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.911193 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686"] Oct 30 18:51:33.916156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.916135 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:33.930970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.930943 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686"] Oct 30 18:51:33.937071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.937046 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-5a90b4\"/\"openshift-service-ca.crt\"" Oct 30 18:51:33.937198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.937078 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-5a90b4\"/\"kube-root-ca.crt\"" Oct 30 18:51:33.937198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.937149 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-5a90b4\"/\"default-dockercfg-bxhmz\"" Oct 30 18:51:33.937389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.937368 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-5a90b4\"/\"e2e-usage-sim-734d96-cert\"" Oct 30 18:51:33.954792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.954763 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-tls\") pod \"e2e-usage-sim-734d96-84d9db57fc-2c686\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:33.955015 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:33.954990 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xq7\" (UniqueName: \"kubernetes.io/projected/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-kube-api-access-d8xq7\") pod \"e2e-usage-sim-734d96-84d9db57fc-2c686\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:34.055907 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:34.055803 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xq7\" (UniqueName: \"kubernetes.io/projected/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-kube-api-access-d8xq7\") pod \"e2e-usage-sim-734d96-84d9db57fc-2c686\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:34.056080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:34.056055 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-tls\") pod \"e2e-usage-sim-734d96-84d9db57fc-2c686\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:34.058599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:34.058580 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-tls\") pod \"e2e-usage-sim-734d96-84d9db57fc-2c686\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:34.072510 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:34.072489 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xq7\" (UniqueName: \"kubernetes.io/projected/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-kube-api-access-d8xq7\") pod \"e2e-usage-sim-734d96-84d9db57fc-2c686\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:34.231012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:34.230979 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:34.536230 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:34.536200 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686"] Oct 30 18:51:34.540388 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:51:34.540357 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0235a09b_b249_4ca9_bbe8_256e83b5cbc1.slice/crio-241736481aeb2927bb3f9950a79903e00051576419d5ea47d7b1e74734e6b6d9 WatchSource:0}: Error finding container 241736481aeb2927bb3f9950a79903e00051576419d5ea47d7b1e74734e6b6d9: Status 404 returned error can't find the container with id 241736481aeb2927bb3f9950a79903e00051576419d5ea47d7b1e74734e6b6d9 Oct 30 18:51:35.159989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:35.159947 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:35.491578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:35.491490 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" event={"ID":"0235a09b-b249-4ca9-bbe8-256e83b5cbc1","Type":"ContainerStarted","Data":"6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6"} Oct 30 18:51:35.491578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:35.491531 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" event={"ID":"0235a09b-b249-4ca9-bbe8-256e83b5cbc1","Type":"ContainerStarted","Data":"241736481aeb2927bb3f9950a79903e00051576419d5ea47d7b1e74734e6b6d9"} Oct 30 18:51:35.491863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:35.491731 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:35.497163 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:35.497136 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:51:35.525913 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:35.525865 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" podStartSLOduration=2.525852127 podStartE2EDuration="2.525852127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:35.522996543 +0000 UTC m=+3192.347390853" watchObservedRunningTime="2026-10-30 18:51:35.525852127 +0000 UTC m=+3192.350246419" Oct 30 18:51:36.018808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:36.018771 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:36.117127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:36.117092 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:36.228241 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:51:36.228185 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:51:36.663539 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:36.663505 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:37.136001 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:37.135969 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:37.598294 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:37.598194 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:37.697642 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:37.697591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:38.002771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:38.002737 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:39.783132 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:39.783092 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:39.946879 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:39.946812 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:40.049352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:40.049263 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:40.767678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:40.767650 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:40.864613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:40.864580 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:42.178232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:42.178199 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:42.286068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:42.286021 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:42.870083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:42.870046 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:44.748571 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:44.748535 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:45.442013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:45.441978 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:45.540178 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:45.540144 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:46.730465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:46.730432 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:48.445692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:48.445658 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:50.182322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:50.182279 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:51.873694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:51.873655 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:53.671598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:53.671563 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:55.922226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:55.922189 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:57.897566 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:57.897532 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:59.680289 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.680257 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2"] Oct 30 18:51:59.686106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.686084 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.703631 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.703605 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2"] Oct 30 18:51:59.718901 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.718876 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8862aa-serving-cert\"" Oct 30 18:51:59.760167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.760134 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:59.821429 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.821398 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-maas-api-tls\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.821593 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.821515 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pvc\" (UniqueName: \"kubernetes.io/projected/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-kube-api-access-57pvc\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.821652 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.821599 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.821703 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.821681 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-metrics-certs\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.842425 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.842398 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:59.922845 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.922777 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-metrics-certs\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.923028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.922964 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-maas-api-tls\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.923125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.923096 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pvc\" (UniqueName: \"kubernetes.io/projected/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-kube-api-access-57pvc\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.923373 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.923350 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.924010 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.923986 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.925899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.925880 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-maas-api-tls\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.931792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.931736 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-metrics-certs\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:51:59.942702 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.942674 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:59.944347 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:51:59.944319 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pvc\" (UniqueName: \"kubernetes.io/projected/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-kube-api-access-57pvc\") pod \"maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:52:00.002658 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:00.002628 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:52:00.307329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:00.307301 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2"] Oct 30 18:52:00.308151 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:52:00.308122 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5456574e_2ce7_4ab4_a0fa_e4e3ba0273f6.slice/crio-00c198a92423780b930031002978938b82741ff38468c9a95ce7d052c6cd187f WatchSource:0}: Error finding container 00c198a92423780b930031002978938b82741ff38468c9a95ce7d052c6cd187f: Status 404 returned error can't find the container with id 00c198a92423780b930031002978938b82741ff38468c9a95ce7d052c6cd187f Oct 30 18:52:00.683343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:00.683314 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" event={"ID":"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6","Type":"ContainerStarted","Data":"00c198a92423780b930031002978938b82741ff38468c9a95ce7d052c6cd187f"} Oct 30 18:52:01.692319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:01.692283 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" event={"ID":"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6","Type":"ContainerStarted","Data":"2da031c1a3008fbf0c7e8c50654458c724f07ae05d74bca2c32964dddbe5a0a8"} Oct 30 18:52:01.692808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:01.692504 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:52:01.730632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:01.730581 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" podStartSLOduration=2.383639193 podStartE2EDuration="2.730568054s" totalImagesPullingTime="346.928861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:01.728981845 +0000 UTC m=+3218.553376159" watchObservedRunningTime="2026-10-30 18:52:01.730568054 +0000 UTC m=+3218.554962346" Oct 30 18:52:01.938213 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:01.938172 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:03.727736 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:03.727698 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:05.456540 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:05.456504 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:06.226850 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:52:06.226800 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:52:07.183309 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:07.183267 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:07.705393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:07.705364 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 18:52:09.034350 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:09.034303 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:10.678238 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:10.678188 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:10.844066 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:10.844019 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:11.350884 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:11.350853 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:12.665588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:12.665552 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:14.471249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:14.471198 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:16.248627 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:16.248576 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:17.549765 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:17.549708 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:17.975666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:17.975627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:19.642083 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:19.642045 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:21.435472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:21.435440 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:23.180997 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:23.180926 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:25.135181 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:25.135151 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:26.965073 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:26.965017 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:27.571562 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:27.571511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:28.733880 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:28.733839 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:28.863972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:28.863930 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:30.478966 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:30.478929 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:32.251582 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:32.251548 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:34.254573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:34.254540 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:36.228215 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:52:36.228172 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:52:36.560285 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:36.560200 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.435497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.435461 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j"] Oct 30 18:52:38.441575 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.441559 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.457633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.457606 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j"] Oct 30 18:52:38.464448 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.464420 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-551d0b-serving-cert\"" Oct 30 18:52:38.534697 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.534667 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.558917 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.558891 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbl2s\" (UniqueName: \"kubernetes.io/projected/3e3b4189-6b83-428d-aa5c-63336edae122-kube-api-access-sbl2s\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.559079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.558968 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-maas-api-tls\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.559079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.559027 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b4189-6b83-428d-aa5c-63336edae122-openshift-service-ca\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.559242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.559222 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-metrics-certs\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.622378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.622354 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.660784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.660745 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-metrics-certs\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.660988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.660883 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbl2s\" (UniqueName: \"kubernetes.io/projected/3e3b4189-6b83-428d-aa5c-63336edae122-kube-api-access-sbl2s\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.660988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.660947 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-maas-api-tls\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.661082 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.661004 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b4189-6b83-428d-aa5c-63336edae122-openshift-service-ca\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.661662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.661637 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b4189-6b83-428d-aa5c-63336edae122-openshift-service-ca\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.663332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.663313 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-maas-api-tls\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.663584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.663563 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-metrics-certs\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.681241 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.681207 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbl2s\" (UniqueName: \"kubernetes.io/projected/3e3b4189-6b83-428d-aa5c-63336edae122-kube-api-access-sbl2s\") pod \"maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:38.757402 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:38.757317 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:39.064770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:39.064741 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j"] Oct 30 18:52:39.066107 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:52:39.066082 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3b4189_6b83_428d_aa5c_63336edae122.slice/crio-eb242cc18f96927c9c34f36888c2cc1383210e601953db4cd3a6b4837a262598 WatchSource:0}: Error finding container eb242cc18f96927c9c34f36888c2cc1383210e601953db4cd3a6b4837a262598: Status 404 returned error can't find the container with id eb242cc18f96927c9c34f36888c2cc1383210e601953db4cd3a6b4837a262598 Oct 30 18:52:39.983613 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:39.983568 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" event={"ID":"3e3b4189-6b83-428d-aa5c-63336edae122","Type":"ContainerStarted","Data":"b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031"} Oct 30 18:52:39.984137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:39.983631 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:39.984137 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:39.983648 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" event={"ID":"3e3b4189-6b83-428d-aa5c-63336edae122","Type":"ContainerStarted","Data":"eb242cc18f96927c9c34f36888c2cc1383210e601953db4cd3a6b4837a262598"} Oct 30 18:52:40.028210 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:40.028147 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" podStartSLOduration=1.65853753 podStartE2EDuration="2.028128631s" totalImagesPullingTime="369.591101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:40.024977174 +0000 UTC m=+3256.849371490" watchObservedRunningTime="2026-10-30 18:52:40.028128631 +0000 UTC m=+3256.852522927" Oct 30 18:52:40.552550 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:40.552510 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:42.451648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:42.451614 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:44.189288 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.189248 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:44.469275 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.469196 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5"] Oct 30 18:52:44.474432 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.474407 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.495017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.494986 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5"] Oct 30 18:52:44.574399 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.574370 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8862aa-openshift-default-dockercfg-5svft\"" Oct 30 18:52:44.637265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637228 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637266 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637359 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637454 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637526 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637551 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxtv\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-kube-api-access-ctxtv\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637631 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed21b2d-f889-45c4-8d56-e5da10476136-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637734 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.637904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.637887 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed21b2d-f889-45c4-8d56-e5da10476136-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739051 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed21b2d-f889-45c4-8d56-e5da10476136-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739112 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739142 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739130 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739165 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739228 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739292 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739453 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739311 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxtv\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-kube-api-access-ctxtv\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739460 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed21b2d-f889-45c4-8d56-e5da10476136-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739565 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739642 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739889 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739697 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.739962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.739939 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed21b2d-f889-45c4-8d56-e5da10476136-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.740165 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.740145 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.740246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.740223 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.741647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.741627 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.741933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.741916 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed21b2d-f889-45c4-8d56-e5da10476136-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.758157 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.758135 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.758246 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.758190 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxtv\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-kube-api-access-ctxtv\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:44.791830 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:44.791799 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:45.107933 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:45.107903 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5"] Oct 30 18:52:45.109564 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:52:45.109526 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed21b2d_f889_45c4_8d56_e5da10476136.slice/crio-886524aaaab745790868b2238f6e7a5c8a56f60d4047f6affc14cbc7d38b1638 WatchSource:0}: Error finding container 886524aaaab745790868b2238f6e7a5c8a56f60d4047f6affc14cbc7d38b1638: Status 404 returned error can't find the container with id 886524aaaab745790868b2238f6e7a5c8a56f60d4047f6affc14cbc7d38b1638 Oct 30 18:52:45.112428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:45.112397 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:52:45.112519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:45.112463 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:52:45.112519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:45.112495 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:52:45.964680 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:45.964641 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:45.997158 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:45.997128 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 18:52:46.026282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:46.026249 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" event={"ID":"bed21b2d-f889-45c4-8d56-e5da10476136","Type":"ContainerStarted","Data":"31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a"} Oct 30 18:52:46.026461 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:46.026296 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" event={"ID":"bed21b2d-f889-45c4-8d56-e5da10476136","Type":"ContainerStarted","Data":"886524aaaab745790868b2238f6e7a5c8a56f60d4047f6affc14cbc7d38b1638"} Oct 30 18:52:46.093588 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:46.093527 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podStartSLOduration=2.093511223 podStartE2EDuration="2.093511223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:46.092629024 +0000 UTC m=+3262.917023366" watchObservedRunningTime="2026-10-30 18:52:46.093511223 +0000 UTC m=+3262.917905515" Oct 30 18:52:46.792315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:46.792280 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:46.793815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:46.793784 4582 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:52:46.793977 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:46.793854 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 18:52:47.763844 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:47.763237 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:47.792520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:47.792486 4582 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:52:47.792681 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:47.792550 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 18:52:48.796221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:48.792351 4582 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:52:48.796221 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:48.792439 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 18:52:49.512203 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:49.512170 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:49.792519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:49.792439 4582 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:52:49.792519 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:49.792502 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 18:52:50.792440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:50.792402 4582 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 18:52:50.792878 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:50.792474 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 18:52:51.369427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:51.369386 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:51.799077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:51.798980 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:51.799594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:51.799570 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:51.801445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:51.801424 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 18:52:52.365920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:52.365886 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:53.155455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:53.155422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:54.969223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:54.969185 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:56.760455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:56.760417 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:58.511280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:52:58.511244 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:00.434568 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:00.434533 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:02.065803 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:02.065767 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:02.184847 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:02.184796 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:02.566351 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:02.566315 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:03.045382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:03.045344 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:03.184996 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:03.184964 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:04.265441 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:04.265399 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:06.086611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:06.086561 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:06.230179 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:53:06.230143 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:53:07.462199 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.462162 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:07.561057 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.561027 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:07.876116 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.876085 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6"] Oct 30 18:53:07.881349 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.881324 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.913988 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.913951 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6"] Oct 30 18:53:07.922249 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922218 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xcs\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-kube-api-access-j7xcs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922394 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922347 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922392 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922463 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922521 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922491 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25eeb43-e27f-48bc-844a-4411c76fed6a-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922561 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922545 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922619 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922604 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922684 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922653 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.922721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.922711 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:07.982262 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:07.982229 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-551d0b-openshift-default-dockercfg-r95tk\"" Oct 30 18:53:08.023828 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.023786 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xcs\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-kube-api-access-j7xcs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.023903 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.023930 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024012 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.023982 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024124 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25eeb43-e27f-48bc-844a-4411c76fed6a-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024207 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024180 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024306 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024362 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024449 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024405 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024504 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024477 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024586 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024801 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.024992 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.024892 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25eeb43-e27f-48bc-844a-4411c76fed6a-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.025068 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.025004 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.026430 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.026387 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.026598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.026583 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.043034 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.043007 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.043126 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.043108 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xcs\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-kube-api-access-j7xcs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.105049 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.105014 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:08.210660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.210561 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:08.532361 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:53:08.532321 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25eeb43_e27f_48bc_844a_4411c76fed6a.slice/crio-a6486666955653ca6756f0f51d4d277ace65e967fe5b6ceb57f6d74f0dcc6813 WatchSource:0}: Error finding container a6486666955653ca6756f0f51d4d277ace65e967fe5b6ceb57f6d74f0dcc6813: Status 404 returned error can't find the container with id a6486666955653ca6756f0f51d4d277ace65e967fe5b6ceb57f6d74f0dcc6813 Oct 30 18:53:08.533455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.533427 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6"] Oct 30 18:53:08.535237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.535191 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:53:08.535354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.535284 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:53:08.535354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:08.535330 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:53:09.229226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:09.229187 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" event={"ID":"f25eeb43-e27f-48bc-844a-4411c76fed6a","Type":"ContainerStarted","Data":"6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd"} Oct 30 18:53:09.229404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:09.229244 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" event={"ID":"f25eeb43-e27f-48bc-844a-4411c76fed6a","Type":"ContainerStarted","Data":"a6486666955653ca6756f0f51d4d277ace65e967fe5b6ceb57f6d74f0dcc6813"} Oct 30 18:53:09.275502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:09.275462 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:09.298164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:09.298110 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podStartSLOduration=2.298096722 podStartE2EDuration="2.298096722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:09.286558545 +0000 UTC m=+3286.110952832" watchObservedRunningTime="2026-10-30 18:53:09.298096722 +0000 UTC m=+3286.122491015" Oct 30 18:53:09.369989 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:09.369952 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:10.211398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:10.211362 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:10.212900 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:10.212871 4582 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:53:10.213035 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:10.212928 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 18:53:10.267899 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:10.267853 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:11.211647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:11.211613 4582 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:53:11.212106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:11.211678 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 18:53:12.100398 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:12.100342 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.202483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:12.202452 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.212033 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:12.211996 4582 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:53:12.212516 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:12.212075 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 18:53:12.785002 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:12.784962 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:13.211542 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:13.211508 4582 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:53:13.211755 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:13.211562 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 18:53:14.068462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:14.068384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:14.211185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:14.211150 4582 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:53:14.211365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:14.211202 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 18:53:15.211032 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:15.211000 4582 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 18:53:15.211496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:15.211052 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 18:53:15.978724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:15.978688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:16.216777 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:16.216751 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:16.217156 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:16.217132 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:16.217833 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:16.217799 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 18:53:17.674891 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:17.674853 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:17.871197 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:17.871162 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:19.538090 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:19.538038 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:21.328670 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:21.328638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:23.101666 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:23.101622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:24.778762 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:24.778723 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:26.424953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:26.424908 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:28.104415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:28.104378 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:28.748905 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:28.748872 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:28.852687 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:28.852644 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:30.750472 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:30.750427 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:32.575270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:32.575236 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:34.362063 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:34.362022 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:36.017752 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:36.017712 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:36.227644 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:53:36.227600 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:53:38.214706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:38.214672 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:39.478834 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:39.478790 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:40.033677 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:40.033643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:41.783459 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:41.783424 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:42.613059 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.613027 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx"] Oct 30 18:53:42.618191 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.618168 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.634741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.634715 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx"] Oct 30 18:53:42.640252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.640229 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1aa59a\"/\"kube-root-ca.crt\"" Oct 30 18:53:42.640252 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.640229 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1aa59a\"/\"openshift-service-ca.crt\"" Oct 30 18:53:42.640455 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.640258 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1aa59a\"/\"default-dockercfg-99hw7\"" Oct 30 18:53:42.645193 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.645173 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1aa59a\"/\"test-model-1aa59a-kserve-self-signed-certs\"" Oct 30 18:53:42.653975 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.653947 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d375b664-fdf7-4634-8a74-495d75055a1c-tls-certs\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.654093 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.654019 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-dshm\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.654219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.654196 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-tmp-dir\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.654274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.654261 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-model-cache\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.654438 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.654415 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2lf\" (UniqueName: \"kubernetes.io/projected/d375b664-fdf7-4634-8a74-495d75055a1c-kube-api-access-vm2lf\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.654614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.654593 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-home\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.662062 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.662040 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:42.756219 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756187 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2lf\" (UniqueName: \"kubernetes.io/projected/d375b664-fdf7-4634-8a74-495d75055a1c-kube-api-access-vm2lf\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.756420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756328 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-home\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.756420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756390 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d375b664-fdf7-4634-8a74-495d75055a1c-tls-certs\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.756553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756450 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-dshm\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.756553 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756537 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-tmp-dir\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.756665 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756588 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-model-cache\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.756792 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756769 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-home\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.756916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756898 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-tmp-dir\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.757009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.756989 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-model-cache\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.758692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.758670 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-dshm\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.758978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.758961 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d375b664-fdf7-4634-8a74-495d75055a1c-tls-certs\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.775268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.775243 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2lf\" (UniqueName: \"kubernetes.io/projected/d375b664-fdf7-4634-8a74-495d75055a1c-kube-api-access-vm2lf\") pod \"test-model-1aa59a-kserve-59b56cbf6c-8c2fx\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:42.934366 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:42.934328 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:43.242663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:43.242632 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx"] Oct 30 18:53:43.243416 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:53:43.243395 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd375b664_fdf7_4634_8a74_495d75055a1c.slice/crio-a4275b6d7167459045824118e350e0c74fd06d43068dffbc6377cde528816bed WatchSource:0}: Error finding container a4275b6d7167459045824118e350e0c74fd06d43068dffbc6377cde528816bed: Status 404 returned error can't find the container with id a4275b6d7167459045824118e350e0c74fd06d43068dffbc6377cde528816bed Oct 30 18:53:43.509107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:43.509033 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" event={"ID":"d375b664-fdf7-4634-8a74-495d75055a1c","Type":"ContainerStarted","Data":"1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26"} Oct 30 18:53:43.509107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:43.509070 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" event={"ID":"d375b664-fdf7-4634-8a74-495d75055a1c","Type":"ContainerStarted","Data":"a4275b6d7167459045824118e350e0c74fd06d43068dffbc6377cde528816bed"} Oct 30 18:53:43.548365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:43.548316 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" podStartSLOduration=1.548302637 podStartE2EDuration="1.548302637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:43.547206007 +0000 UTC m=+3320.371600299" watchObservedRunningTime="2026-10-30 18:53:43.548302637 +0000 UTC m=+3320.372696928" Oct 30 18:53:43.591742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:43.591710 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:43.935382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:43.935350 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:45.367531 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:45.367501 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:47.184529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:47.184495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:49.403789 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:49.403748 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:51.466663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:51.466622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:52.935357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:52.935328 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:52.948276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:52.948249 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:53.220971 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:53.220893 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:53.591569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:53.591493 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 18:53:54.940313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:54.940277 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:55.042713 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:55.042662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:56.861867 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:56.861806 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:58.639431 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:53:58.639372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:00.456008 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:00.455973 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:02.172125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:02.172088 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:02.637498 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:02.637443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:02.736143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:02.736111 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:03.304460 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:03.304429 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:03.565136 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:03.565045 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:04.654403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:04.654352 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:06.227554 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:54:06.227518 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:54:06.420047 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:06.420013 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:06.545696 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:06.545529 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:07.205408 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:07.205372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:08.691578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:08.691542 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:09.901209 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:09.901175 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:10.010673 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:10.010622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:10.405660 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:10.405623 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:10.506547 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:10.506511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:10.917481 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:10.917441 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:11.027589 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:11.027560 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:12.599929 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:12.599889 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:14.376486 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:14.376451 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:14.858554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:14.858466 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:14.965167 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:14.965135 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:16.189420 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:16.189391 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:16.311503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:16.311391 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:16.438497 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:16.438455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:16.805145 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:16.805114 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686"] Oct 30 18:54:16.805313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:16.805259 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" podUID="0235a09b-b249-4ca9-bbe8-256e83b5cbc1" containerName="sim" containerID="cri-o://6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6" gracePeriod=30 Oct 30 18:54:17.017584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.017473 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:17.161802 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.161775 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:54:17.267843 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.267785 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-tls\" (UniqueName: \"kubernetes.io/secret/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-tls\") pod \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " Oct 30 18:54:17.268282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.267971 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-kube-api-access-d8xq7\" (UniqueName: \"kubernetes.io/projected/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-kube-api-access-d8xq7\") pod \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\" (UID: \"0235a09b-b249-4ca9-bbe8-256e83b5cbc1\") " Oct 30 18:54:17.270409 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.270377 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-tls" pod "0235a09b-b249-4ca9-bbe8-256e83b5cbc1" (UID: "0235a09b-b249-4ca9-bbe8-256e83b5cbc1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:17.270537 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.270464 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-kube-api-access-d8xq7" pod "0235a09b-b249-4ca9-bbe8-256e83b5cbc1" (UID: "0235a09b-b249-4ca9-bbe8-256e83b5cbc1"). InnerVolumeSpecName "kube-api-access-d8xq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:17.370054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.369953 4582 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:54:17.370054 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.369990 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xq7\" (UniqueName: \"kubernetes.io/projected/0235a09b-b249-4ca9-bbe8-256e83b5cbc1-kube-api-access-d8xq7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:54:17.783603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.783523 4582 generic.go:356] "Generic (PLEG): container finished" podID="0235a09b-b249-4ca9-bbe8-256e83b5cbc1" containerID="6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6" exitCode=0 Oct 30 18:54:17.783603 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.783568 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" Oct 30 18:54:17.783804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.783609 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" event={"ID":"0235a09b-b249-4ca9-bbe8-256e83b5cbc1","Type":"ContainerDied","Data":"6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6"} Oct 30 18:54:17.783804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.783655 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686" event={"ID":"0235a09b-b249-4ca9-bbe8-256e83b5cbc1","Type":"ContainerDied","Data":"241736481aeb2927bb3f9950a79903e00051576419d5ea47d7b1e74734e6b6d9"} Oct 30 18:54:17.783804 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.783677 4582 scope.go:122] "RemoveContainer" containerID="6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6" Oct 30 18:54:17.794714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.794694 4582 scope.go:122] "RemoveContainer" containerID="6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6" Oct 30 18:54:17.795026 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:54:17.794992 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6\": container with ID starting with 6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6 not found: ID does not exist" containerID="6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6" Oct 30 18:54:17.795108 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.795030 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6"} err="failed to get container status \"6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6\": rpc error: code = NotFound desc = could not find container \"6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6\": container with ID starting with 6b3dce63fc340c1d842f3675a758d3963f8408f67b57c2e26e15c831cb3175b6 not found: ID does not exist" Oct 30 18:54:17.809618 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.809598 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:17.833123 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.833098 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686"] Oct 30 18:54:17.848017 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:17.847979 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-5a90b4/e2e-usage-sim-734d96-84d9db57fc-2c686"] Oct 30 18:54:18.351315 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:18.351276 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:18.449107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:18.449075 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:19.301182 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:19.301144 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:19.302592 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:19.302571 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0235a09b-b249-4ca9-bbe8-256e83b5cbc1" path="/var/lib/kubelet/pods/0235a09b-b249-4ca9-bbe8-256e83b5cbc1/volumes" Oct 30 18:54:21.114153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:21.114119 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.961478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:22.961444 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:24.786357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:24.786322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:26.600951 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:26.600906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:28.252811 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:28.252774 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:30.052742 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:30.052701 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:31.730910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:31.730872 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:33.497270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:33.497235 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:35.376647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:35.376611 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:36.237270 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:54:36.237231 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:54:38.943831 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:38.943780 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:40.416283 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:40.416241 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:40.516104 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:40.516061 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:41.471640 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:41.471611 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:42.143721 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:42.143680 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:42.247174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:42.247134 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:42.333959 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:42.333927 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:42.800609 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:42.800576 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:43.139422 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:43.139373 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:44.048815 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:44.048775 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:45.874650 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:45.874620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:47.659695 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:47.659655 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:48.653876 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:48.653844 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:48.761554 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:48.761506 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:49.201625 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:49.201590 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:49.357541 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:49.357507 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:51.044506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:51.044474 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:52.755758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:52.755662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:54.570161 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:54.570108 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:56.359692 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:56.359638 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:58.337598 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:58.337557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:59.678864 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:59.678785 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:59.784290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:54:59.784252 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:00.620924 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:00.620886 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:02.356750 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:02.356716 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:03.634849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:03.634798 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:03.735483 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:03.735444 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:03.838150 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:03.838121 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:03.947251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:03.947161 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:04.316500 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:04.316425 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:05.555470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:05.555433 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:05.657663 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:05.657621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:06.226763 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:55:06.226721 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:55:06.509709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:06.509624 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:08.218791 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:08.218751 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:10.146785 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:10.146749 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:12.177770 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:12.177739 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:14.385788 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:14.385751 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:15.111458 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:15.111423 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:16.314223 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.314177 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:16.407381 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.407346 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m"] Oct 30 18:55:16.408183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.408161 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0235a09b-b249-4ca9-bbe8-256e83b5cbc1" containerName="sim" Oct 30 18:55:16.408183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.408177 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0235a09b-b249-4ca9-bbe8-256e83b5cbc1" containerName="sim" Oct 30 18:55:16.408389 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.408333 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0235a09b-b249-4ca9-bbe8-256e83b5cbc1" containerName="sim" Oct 30 18:55:16.412006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.411983 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.440471 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.440438 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m"] Oct 30 18:55:16.445882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.445839 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8862aa\"/\"kube-root-ca.crt\"" Oct 30 18:55:16.446021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.445917 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8862aa\"/\"test-model-8862aa-kserve-self-signed-certs\"" Oct 30 18:55:16.446021 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.445973 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8862aa\"/\"default-dockercfg-qnsz6\"" Oct 30 18:55:16.446237 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.446214 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8862aa\"/\"openshift-service-ca.crt\"" Oct 30 18:55:16.469870 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.469842 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:16.517694 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.517662 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tmp-dir\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.517886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.517766 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9rp\" (UniqueName: \"kubernetes.io/projected/f5581b97-02bd-4aa6-b85c-c1ef2813795f-kube-api-access-pl9rp\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.517886 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.517858 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-dshm\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.517980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.517912 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-model-cache\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.518028 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.518013 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tls-certs\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.518076 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.518037 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-home\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619183 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619158 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-model-cache\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619227 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tls-certs\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619246 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-home\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619348 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619340 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tmp-dir\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619393 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9rp\" (UniqueName: \"kubernetes.io/projected/f5581b97-02bd-4aa6-b85c-c1ef2813795f-kube-api-access-pl9rp\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619473 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619420 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-dshm\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619622 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619599 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-model-cache\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619619 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-home\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.619760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.619734 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tmp-dir\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.621569 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.621549 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-dshm\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.621648 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.621582 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tls-certs\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.638615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.638559 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9rp\" (UniqueName: \"kubernetes.io/projected/f5581b97-02bd-4aa6-b85c-c1ef2813795f-kube-api-access-pl9rp\") pod \"test-model-8862aa-kserve-59d8b5c5-75c7m\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:16.727635 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:16.727607 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:17.066052 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:17.066028 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m"] Oct 30 18:55:17.067114 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:55:17.067083 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5581b97_02bd_4aa6_b85c_c1ef2813795f.slice/crio-4c81d114caeac1fbb88874b9108c739c3404cc8bd8c7e5fd3c153649b767dcd5 WatchSource:0}: Error finding container 4c81d114caeac1fbb88874b9108c739c3404cc8bd8c7e5fd3c153649b767dcd5: Status 404 returned error can't find the container with id 4c81d114caeac1fbb88874b9108c739c3404cc8bd8c7e5fd3c153649b767dcd5 Oct 30 18:55:17.235406 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:17.235367 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" event={"ID":"f5581b97-02bd-4aa6-b85c-c1ef2813795f","Type":"ContainerStarted","Data":"f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114"} Oct 30 18:55:17.235599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:17.235424 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" event={"ID":"f5581b97-02bd-4aa6-b85c-c1ef2813795f","Type":"ContainerStarted","Data":"4c81d114caeac1fbb88874b9108c739c3404cc8bd8c7e5fd3c153649b767dcd5"} Oct 30 18:55:17.287492 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:17.287435 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" podStartSLOduration=1.28742279 podStartE2EDuration="1.28742279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:17.283479967 +0000 UTC m=+3414.107874258" watchObservedRunningTime="2026-10-30 18:55:17.28742279 +0000 UTC m=+3414.111817081" Oct 30 18:55:17.728265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:17.728225 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:17.910445 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:17.910412 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:18.148885 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.148851 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv"] Oct 30 18:55:18.153998 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.153976 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.179897 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.179868 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv"] Oct 30 18:55:18.180140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.180119 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2820d9-serving-cert\"" Oct 30 18:55:18.240413 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.240383 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.240591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.240441 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.240591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.240496 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-metrics-certs\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.240591 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.240576 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46tk\" (UniqueName: \"kubernetes.io/projected/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-kube-api-access-f46tk\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.341559 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.341517 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:18.341780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.341570 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.341780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.341665 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.341780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.341685 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-metrics-certs\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.341780 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.341726 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46tk\" (UniqueName: \"kubernetes.io/projected/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-kube-api-access-f46tk\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.342397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.342371 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.344143 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.344125 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-metrics-certs\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.344581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.344560 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.361855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.360930 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46tk\" (UniqueName: \"kubernetes.io/projected/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-kube-api-access-f46tk\") pod \"maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.423580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.423506 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:18.487080 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.487051 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:18.796580 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.796307 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv"] Oct 30 18:55:18.810857 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:55:18.810797 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b904d1_097d_4a1a_a5d6_6367b4a426dd.slice/crio-b202ab7d6959da70df5ccd7e6a6562ec66f08d090729971b1963479d3bf16ae0 WatchSource:0}: Error finding container b202ab7d6959da70df5ccd7e6a6562ec66f08d090729971b1963479d3bf16ae0: Status 404 returned error can't find the container with id b202ab7d6959da70df5ccd7e6a6562ec66f08d090729971b1963479d3bf16ae0 Oct 30 18:55:18.812978 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:18.812954 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:55:19.253894 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:19.253856 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" event={"ID":"b2b904d1-097d-4a1a-a5d6-6367b4a426dd","Type":"ContainerStarted","Data":"dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95"} Oct 30 18:55:19.254025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:19.253917 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" event={"ID":"b2b904d1-097d-4a1a-a5d6-6367b4a426dd","Type":"ContainerStarted","Data":"b202ab7d6959da70df5ccd7e6a6562ec66f08d090729971b1963479d3bf16ae0"} Oct 30 18:55:19.254115 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:19.254097 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:19.293204 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:19.293081 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" podStartSLOduration=0.936623463 podStartE2EDuration="1.293063449s" totalImagesPullingTime="356.439986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:19.2920885 +0000 UTC m=+3416.116482812" watchObservedRunningTime="2026-10-30 18:55:19.293063449 +0000 UTC m=+3416.117457742" Oct 30 18:55:20.268706 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:20.268671 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:21.977614 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:21.977573 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:23.646217 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:23.646182 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:25.268159 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:25.268131 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 18:55:25.468506 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:25.468459 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:26.728581 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:26.728547 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:26.744085 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:26.744058 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:27.335621 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:27.335589 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 18:55:27.869586 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:27.869543 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:29.769071 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:29.769026 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:31.666077 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:31.666044 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:33.365299 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:33.365268 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:35.197019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:35.196981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:36.227187 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:55:36.227152 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:55:36.954361 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:36.954329 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:38.623725 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:38.623692 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:40.257724 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:40.257688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:40.426662 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:40.426625 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:42.473574 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:42.473533 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:44.291428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:44.291399 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:46.078105 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:46.078066 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:48.017969 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:48.017933 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:49.674661 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:49.674586 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:51.552602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:51.552566 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:52.074245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:52.074206 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:53.278378 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:53.278336 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:55.237222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:55.237190 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:57.650741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:57.650703 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:59.462549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:55:59.462514 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:01.291329 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:01.291294 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:03.098270 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:03.098236 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:04.837608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:04.837567 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:04.990629 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:04.990592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:06.232704 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:56:06.232668 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:56:06.604693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.604618 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8"] Oct 30 18:56:06.611179 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.611151 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.630513 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.630483 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8"] Oct 30 18:56:06.658948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.658914 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-869b55-serving-cert\"" Oct 30 18:56:06.688280 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.688242 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:06.742004 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.741965 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptc9\" (UniqueName: \"kubernetes.io/projected/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-kube-api-access-5ptc9\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.742175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.742030 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-openshift-service-ca\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.742175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.742134 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-metrics-certs\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.742463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.742442 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-maas-api-tls\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.788013 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.787976 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:06.844140 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.844101 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-maas-api-tls\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.844296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.844154 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptc9\" (UniqueName: \"kubernetes.io/projected/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-kube-api-access-5ptc9\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.844296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.844190 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-openshift-service-ca\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.844296 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.844247 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-metrics-certs\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.846848 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.846806 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-metrics-certs\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.847515 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.847488 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-maas-api-tls\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.847664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.847646 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-openshift-service-ca\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.865986 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.865931 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptc9\" (UniqueName: \"kubernetes.io/projected/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-kube-api-access-5ptc9\") pod \"maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:06.911265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.911239 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:06.929863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:06.928095 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:07.264379 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:07.264355 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8"] Oct 30 18:56:07.265049 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:56:07.265021 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b63d395_5cd1_4b4a_82d9_3d33b04fc502.slice/crio-e263e018ad5f589f28050b9deca2a5eda4d759f751f8e548aaad22141d1eedc7 WatchSource:0}: Error finding container e263e018ad5f589f28050b9deca2a5eda4d759f751f8e548aaad22141d1eedc7: Status 404 returned error can't find the container with id e263e018ad5f589f28050b9deca2a5eda4d759f751f8e548aaad22141d1eedc7 Oct 30 18:56:07.631442 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:07.631391 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" event={"ID":"9b63d395-5cd1-4b4a-82d9-3d33b04fc502","Type":"ContainerStarted","Data":"e263e018ad5f589f28050b9deca2a5eda4d759f751f8e548aaad22141d1eedc7"} Oct 30 18:56:08.639952 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:08.639911 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" event={"ID":"9b63d395-5cd1-4b4a-82d9-3d33b04fc502","Type":"ContainerStarted","Data":"68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e"} Oct 30 18:56:08.640358 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:08.640022 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:08.684436 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:08.684386 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" podStartSLOduration=2.299155385 podStartE2EDuration="2.684372216s" totalImagesPullingTime="385.216831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:08.680038321 +0000 UTC m=+3465.504432632" watchObservedRunningTime="2026-10-30 18:56:08.684372216 +0000 UTC m=+3465.508766507" Oct 30 18:56:08.722446 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:08.722412 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:10.662941 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:10.662901 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:12.371741 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:12.371706 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:14.086857 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:14.086797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:14.652850 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:14.652783 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 18:56:15.901961 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:15.901926 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:17.673202 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:17.673167 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:18.026980 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:18.026898 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:19.640740 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:19.640705 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:21.362727 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:21.362697 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:23.638058 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:23.638003 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:25.586953 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:25.586915 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:27.451608 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:27.451572 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:29.191462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:29.191428 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:30.990311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:30.990280 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:32.775127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:32.775095 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:34.669187 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:34.669139 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:36.226493 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:56:36.226448 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:56:36.388849 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:36.388789 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:38.195079 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:38.195036 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:38.432689 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:38.432649 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:38.636310 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:38.636278 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:38.954705 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:38.954626 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:39.206416 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:39.206322 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:39.355235 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:39.355198 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:40.173370 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:40.173336 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:40.965152 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:40.965115 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:42.916277 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:42.916242 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:43.374606 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:43.374570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:43.520733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:43.520699 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:45.206418 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:45.206377 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:45.357950 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:45.357914 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:45.511433 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:45.511352 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:47.202512 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:47.202470 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:48.974355 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:48.974321 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:50.868265 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:50.868233 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:52.531084 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:52.531051 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:54.276042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:54.276006 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:56.035493 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:56.035463 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:57.344261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:57.344227 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2"] Oct 30 18:56:57.344733 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:57.344482 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" containerID="cri-o://aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625" gracePeriod=30 Oct 30 18:56:57.788375 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:57.788341 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:59.690717 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:56:59.690678 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:01.385567 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:01.385534 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:01.600808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:01.600771 4582 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:57:01.601009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:01.600853 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:57:02.630970 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.630947 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:57:02.756758 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.756730 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-envoy\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.756960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.756768 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-certs\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.756960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.756908 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-credential-socket\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.756960 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.756930 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-podinfo\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.757174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757038 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-kube-api-access-46xpr\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-kube-api-access-46xpr\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.757174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757088 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istiod-ca-cert\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.757174 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757131 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-certs" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.757365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757232 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-token\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-token\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.757365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757234 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-credential-socket" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.757491 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757452 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istiod-ca-cert" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:02.757558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757492 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-socket\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.757558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757524 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-data\") pod \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\" (UID: \"5ad432de-0709-43a2-8eb1-cc5abd5c300a\") " Oct 30 18:57:02.757773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757749 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-socket" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.757931 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.757874 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-data" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.758266 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.758247 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.758319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.758271 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.758319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.758285 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.758319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.758299 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.758319 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.758312 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.759578 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.759552 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-envoy" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.759903 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.759881 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-podinfo" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:57:02.760006 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.759900 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-token" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.760367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.760346 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-kube-api-access-46xpr" pod "5ad432de-0709-43a2-8eb1-cc5abd5c300a" (UID: "5ad432de-0709-43a2-8eb1-cc5abd5c300a"). InnerVolumeSpecName "kube-api-access-46xpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.860248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.860208 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xpr\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-kube-api-access-46xpr\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.860248 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.860247 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.860450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.860262 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.860450 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:02.860279 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad432de-0709-43a2-8eb1-cc5abd5c300a-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:57:03.042065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.041978 4582 generic.go:356] "Generic (PLEG): container finished" podID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerID="aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625" exitCode=0 Oct 30 18:57:03.042065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.042013 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" event={"ID":"5ad432de-0709-43a2-8eb1-cc5abd5c300a","Type":"ContainerDied","Data":"aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625"} Oct 30 18:57:03.042065 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.042028 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" Oct 30 18:57:03.042340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.042073 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2" event={"ID":"5ad432de-0709-43a2-8eb1-cc5abd5c300a","Type":"ContainerDied","Data":"0f3c96ffaf48dcd0af92df0dc4e2acdb120e4efa63d6854c25a542c06fb6d11f"} Oct 30 18:57:03.042340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.042101 4582 scope.go:122] "RemoveContainer" containerID="aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625" Oct 30 18:57:03.053910 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.053891 4582 scope.go:122] "RemoveContainer" containerID="aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625" Oct 30 18:57:03.054187 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:57:03.054166 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625\": container with ID starting with aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625 not found: ID does not exist" containerID="aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625" Oct 30 18:57:03.054263 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.054198 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625"} err="failed to get container status \"aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625\": rpc error: code = NotFound desc = could not find container \"aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625\": container with ID starting with aa67224ec3a5845b453cd92b3cc359d5c5ce2bdf24d6a00d938054c1c797e625 not found: ID does not exist" Oct 30 18:57:03.101386 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.101359 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2"] Oct 30 18:57:03.108354 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.108328 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vfhk2"] Oct 30 18:57:03.212226 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.212190 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:03.320656 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:03.320567 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" path="/var/lib/kubelet/pods/5ad432de-0709-43a2-8eb1-cc5abd5c300a/volumes" Oct 30 18:57:05.078198 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:05.078168 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:05.207753 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:05.207721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:05.913974 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:05.913941 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:06.227394 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:57:06.227318 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:57:06.873276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:06.873243 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:08.663605 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:08.663561 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:10.472882 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:10.472846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:12.756743 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:12.756708 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:14.658778 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:14.658744 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:16.578520 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:16.578437 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:17.328397 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:17.328361 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:17.855573 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:17.855531 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:18.228647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:18.228615 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:18.415229 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:18.415196 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:18.527632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:18.527549 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:18.628533 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:18.628443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:20.485711 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:20.485682 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:22.364323 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:22.364285 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:24.188655 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:24.188620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:25.984958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:25.984918 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:27.913232 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:27.913199 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:29.686787 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:29.686750 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:31.570940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:31.570857 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:33.424470 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:33.424431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:35.129091 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:35.129046 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:36.226805 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:57:36.226771 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:57:36.998317 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:36.998277 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:38.769325 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:38.769288 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:40.507311 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:40.507279 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:42.234415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:42.234387 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:44.084250 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:44.084217 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:45.734468 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.734437 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6"] Oct 30 18:57:45.735287 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.735270 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" Oct 30 18:57:45.735343 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.735288 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" Oct 30 18:57:45.735440 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.735431 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad432de-0709-43a2-8eb1-cc5abd5c300a" containerName="istio-proxy" Oct 30 18:57:45.739451 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.739431 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.758332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.758308 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6"] Oct 30 18:57:45.765558 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.765529 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9f6b7c-serving-cert\"" Oct 30 18:57:45.813324 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.813292 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:45.879274 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.879231 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dww8d\" (UniqueName: \"kubernetes.io/projected/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-kube-api-access-dww8d\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.879502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.879339 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.879502 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.879404 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-maas-api-tls\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.879611 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.879595 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-metrics-certs\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.913503 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.913472 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:45.980623 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.980583 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-metrics-certs\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.980808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.980675 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dww8d\" (UniqueName: \"kubernetes.io/projected/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-kube-api-access-dww8d\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.980808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.980721 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.980808 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.980762 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-maas-api-tls\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.981478 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.981451 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.983173 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.983153 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-metrics-certs\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.983284 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.983194 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-maas-api-tls\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:45.999401 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:45.999336 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dww8d\" (UniqueName: \"kubernetes.io/projected/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-kube-api-access-dww8d\") pod \"maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:46.026415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:46.026376 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:46.055599 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:46.055563 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:46.365647 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:46.365617 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6"] Oct 30 18:57:46.367052 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:57:46.367021 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53c015c_5a4f_4d06_95dd_5ff15c9e225f.slice/crio-2b4c94b8ffdf85c431f7f2d10c47a4054655b3a1b8179664ff10ffc3ca3b0e6e WatchSource:0}: Error finding container 2b4c94b8ffdf85c431f7f2d10c47a4054655b3a1b8179664ff10ffc3ca3b0e6e: Status 404 returned error can't find the container with id 2b4c94b8ffdf85c431f7f2d10c47a4054655b3a1b8179664ff10ffc3ca3b0e6e Oct 30 18:57:47.377545 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:47.377513 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" event={"ID":"f53c015c-5a4f-4d06-95dd-5ff15c9e225f","Type":"ContainerStarted","Data":"d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e"} Oct 30 18:57:47.378029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:47.377564 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" event={"ID":"f53c015c-5a4f-4d06-95dd-5ff15c9e225f","Type":"ContainerStarted","Data":"2b4c94b8ffdf85c431f7f2d10c47a4054655b3a1b8179664ff10ffc3ca3b0e6e"} Oct 30 18:57:47.378029 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:47.377605 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:47.428881 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:47.428805 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" podStartSLOduration=1.9875489499999999 podStartE2EDuration="2.428787503s" totalImagesPullingTime="441.238553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:47.426702736 +0000 UTC m=+3564.251097029" watchObservedRunningTime="2026-10-30 18:57:47.428787503 +0000 UTC m=+3564.253181795" Oct 30 18:57:47.971146 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:47.971112 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:48.084962 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:48.084921 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.137949 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:50.137912 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.763669 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:50.763639 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.878176 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:50.878142 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:52.178585 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:52.178544 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:52.617462 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:52.617383 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:52.720549 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:52.720515 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:53.390022 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:53.389990 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 18:57:54.174535 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:54.174489 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:56.103261 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:56.103215 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:57.985268 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:57.985227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:59.878101 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:57:59.878063 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:01.635069 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:01.635032 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:03.480529 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:03.480492 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:05.271584 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:05.271553 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:06.226938 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:58:06.226892 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:58:07.050307 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:07.050265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:09.152489 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:09.152443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:09.947427 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:09.947397 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:11.235258 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:11.235219 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:13.042860 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:13.042806 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:14.904715 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:14.904682 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:16.680404 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:16.680368 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:18.264830 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:18.264796 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:18.454415 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:18.454384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:20.421626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:20.421588 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:21.116915 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:21.116875 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:21.334367 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:21.334327 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:22.593019 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:22.592963 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:24.747594 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:24.747552 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:26.612909 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:26.612877 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:28.611306 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:28.611264 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:30.440428 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:30.440394 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:32.273507 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:32.273477 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:33.954271 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:33.954238 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:35.803678 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:35.803645 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:36.226721 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:58:36.226680 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:58:37.785972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:37.785938 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:39.704612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:39.704574 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:41.567726 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:41.567693 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:43.243106 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:43.243075 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:44.925855 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:44.925799 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:46.680148 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:46.680116 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:48.444125 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:48.444094 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:50.153714 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:50.153686 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:52.066259 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:52.066227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:53.852551 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:53.852512 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.763709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.763676 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6"] Oct 30 18:58:55.768732 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.768710 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.786175 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.786147 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6"] Oct 30 18:58:55.793935 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.793891 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-daf3ba-serving-cert\"" Oct 30 18:58:55.841945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.841897 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.870172 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.870142 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-maas-api-tls\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.870352 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.870257 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c596df49-8d89-4082-a5d2-d8f720f8c584-openshift-service-ca\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.870403 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.870381 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-metrics-certs\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.870534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.870510 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnspt\" (UniqueName: \"kubernetes.io/projected/c596df49-8d89-4082-a5d2-d8f720f8c584-kube-api-access-qnspt\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.934160 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.934129 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.971737 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.971697 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-metrics-certs\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.971957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.971863 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnspt\" (UniqueName: \"kubernetes.io/projected/c596df49-8d89-4082-a5d2-d8f720f8c584-kube-api-access-qnspt\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.972036 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.971978 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-maas-api-tls\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.972107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.972056 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c596df49-8d89-4082-a5d2-d8f720f8c584-openshift-service-ca\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.972784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.972755 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c596df49-8d89-4082-a5d2-d8f720f8c584-openshift-service-ca\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.974365 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.974341 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-maas-api-tls\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.974454 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.974385 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-metrics-certs\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:55.991556 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:55.991531 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnspt\" (UniqueName: \"kubernetes.io/projected/c596df49-8d89-4082-a5d2-d8f720f8c584-kube-api-access-qnspt\") pod \"maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:56.037779 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.037684 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:56.084615 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.084581 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:56.275282 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.275248 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:56.392196 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.392168 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6"] Oct 30 18:58:56.392507 ip-10-0-133-124 kubenswrapper[4582]: W1030 18:58:56.392483 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc596df49_8d89_4082_a5d2_d8f720f8c584.slice/crio-fe67e4b0f83f6c2e1c837124d390d9b84c061dfcc42c74f4870dd1e46fcd8718 WatchSource:0}: Error finding container fe67e4b0f83f6c2e1c837124d390d9b84c061dfcc42c74f4870dd1e46fcd8718: Status 404 returned error can't find the container with id fe67e4b0f83f6c2e1c837124d390d9b84c061dfcc42c74f4870dd1e46fcd8718 Oct 30 18:58:56.909807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.909766 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" event={"ID":"c596df49-8d89-4082-a5d2-d8f720f8c584","Type":"ContainerStarted","Data":"012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc"} Oct 30 18:58:56.909807 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.909847 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" event={"ID":"c596df49-8d89-4082-a5d2-d8f720f8c584","Type":"ContainerStarted","Data":"fe67e4b0f83f6c2e1c837124d390d9b84c061dfcc42c74f4870dd1e46fcd8718"} Oct 30 18:58:56.910313 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.909881 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:58:56.953185 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:56.953137 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" podStartSLOduration=1.603980431 podStartE2EDuration="1.95312305s" totalImagesPullingTime="349.142619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:56.948400824 +0000 UTC m=+3633.772795116" watchObservedRunningTime="2026-10-30 18:58:56.95312305 +0000 UTC m=+3633.777517341" Oct 30 18:58:57.811095 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:57.811065 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:58.118218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:58.118171 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:59.964659 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:58:59.964625 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:01.775435 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:01.775398 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:02.923382 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:02.923349 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 18:59:03.680590 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:03.680558 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:05.481011 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:05.480970 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:06.226896 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:59:06.226855 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:59:07.175228 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:07.175196 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:08.943218 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:08.943184 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:10.770251 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:10.770203 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:12.668602 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:12.668566 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:14.398245 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:14.398199 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:16.177025 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:16.176995 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:17.968100 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:17.968067 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:18.363426 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:18.363328 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:19.693037 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:19.692998 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:21.459546 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:21.459513 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:22.815940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:22.815897 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:23.515030 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:23.514997 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:25.359693 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:25.359658 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:26.848009 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:26.847974 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h"] Oct 30 18:59:26.848536 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:26.848165 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" podUID="0cb1fe32-df10-442d-917f-090ee73cfb32" containerName="maas-api" containerID="cri-o://6752affe7d655463946bd1aa312bfd780f3cb7e6e88391a4385f6b6bbfb83b60" gracePeriod=30 Oct 30 18:59:26.900583 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:26.900513 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:27.148781 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.148747 4582 generic.go:356] "Generic (PLEG): container finished" podID="0cb1fe32-df10-442d-917f-090ee73cfb32" containerID="6752affe7d655463946bd1aa312bfd780f3cb7e6e88391a4385f6b6bbfb83b60" exitCode=0 Oct 30 18:59:27.148957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.148860 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" event={"ID":"0cb1fe32-df10-442d-917f-090ee73cfb32","Type":"ContainerDied","Data":"6752affe7d655463946bd1aa312bfd780f3cb7e6e88391a4385f6b6bbfb83b60"} Oct 30 18:59:27.148957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.148922 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" event={"ID":"0cb1fe32-df10-442d-917f-090ee73cfb32","Type":"ContainerDied","Data":"71104002ddc02fa413bdd7ed5d91a827975fd5d2a3d5145b38ffbd82f473b3d6"} Oct 30 18:59:27.148957 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.148941 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71104002ddc02fa413bdd7ed5d91a827975fd5d2a3d5145b38ffbd82f473b3d6" Oct 30 18:59:27.174920 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.174895 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:59:27.214754 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.214717 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-metrics-certs\") pod \"0cb1fe32-df10-442d-917f-090ee73cfb32\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " Oct 30 18:59:27.214948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.214770 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb1fe32-df10-442d-917f-090ee73cfb32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb1fe32-df10-442d-917f-090ee73cfb32-openshift-service-ca\") pod \"0cb1fe32-df10-442d-917f-090ee73cfb32\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " Oct 30 18:59:27.214948 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.214882 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-maas-api-tls\") pod \"0cb1fe32-df10-442d-917f-090ee73cfb32\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " Oct 30 18:59:27.215086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.214981 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb1fe32-df10-442d-917f-090ee73cfb32-kube-api-access-ws4g2\" (UniqueName: \"kubernetes.io/projected/0cb1fe32-df10-442d-917f-090ee73cfb32-kube-api-access-ws4g2\") pod \"0cb1fe32-df10-442d-917f-090ee73cfb32\" (UID: \"0cb1fe32-df10-442d-917f-090ee73cfb32\") " Oct 30 18:59:27.215773 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.215739 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb1fe32-df10-442d-917f-090ee73cfb32-openshift-service-ca" pod "0cb1fe32-df10-442d-917f-090ee73cfb32" (UID: "0cb1fe32-df10-442d-917f-090ee73cfb32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:27.218086 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.218057 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb1fe32-df10-442d-917f-090ee73cfb32-kube-api-access-ws4g2" pod "0cb1fe32-df10-442d-917f-090ee73cfb32" (UID: "0cb1fe32-df10-442d-917f-090ee73cfb32"). InnerVolumeSpecName "kube-api-access-ws4g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:27.218257 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.218240 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-maas-api-tls" pod "0cb1fe32-df10-442d-917f-090ee73cfb32" (UID: "0cb1fe32-df10-442d-917f-090ee73cfb32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:27.218340 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.218322 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-metrics-certs" pod "0cb1fe32-df10-442d-917f-090ee73cfb32" (UID: "0cb1fe32-df10-442d-917f-090ee73cfb32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:27.316981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.316949 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:27.316981 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.316979 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb1fe32-df10-442d-917f-090ee73cfb32-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:27.317155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.316989 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb1fe32-df10-442d-917f-090ee73cfb32-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:27.317155 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.316999 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws4g2\" (UniqueName: \"kubernetes.io/projected/0cb1fe32-df10-442d-917f-090ee73cfb32-kube-api-access-ws4g2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:27.812222 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:27.812185 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:28.021164 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:28.021131 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:28.157255 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:28.157222 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h" Oct 30 18:59:28.201916 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:28.201882 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:28.213617 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:28.213585 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h"] Oct 30 18:59:28.217290 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:28.217263 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0cc6b1-79dd7ddbdb-sm67h"] Oct 30 18:59:28.286865 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:28.286836 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:28.609757 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:28.609680 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:29.302457 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:29.302426 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb1fe32-df10-442d-917f-090ee73cfb32" path="/var/lib/kubelet/pods/0cb1fe32-df10-442d-917f-090ee73cfb32/volumes" Oct 30 18:59:30.180127 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:30.180095 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:31.321465 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:31.321428 4582 scope.go:122] "RemoveContainer" containerID="6752affe7d655463946bd1aa312bfd780f3cb7e6e88391a4385f6b6bbfb83b60" Oct 30 18:59:31.967612 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:31.967581 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:33.712322 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:33.712288 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:35.169945 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:35.169902 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:35.275958 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:35.275915 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:35.551042 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:35.550960 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:36.230382 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:59:36.230345 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 18:59:36.833633 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:36.833598 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:38.758479 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:38.758439 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:40.602707 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:40.602670 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:42.668863 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:42.668800 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:44.471107 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:44.471077 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:46.149639 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:46.149601 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:47.031463 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:47.031424 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:47.140078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:47.139959 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:47.816192 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:47.816159 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr"] Oct 30 18:59:47.816626 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:47.816362 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" podUID="0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" containerName="maas-api" containerID="cri-o://281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f" gracePeriod=30 Oct 30 18:59:47.875716 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:47.875678 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:48.099908 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.099884 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:59:48.219393 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.219358 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-openshift-service-ca\") pod \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " Oct 30 18:59:48.219616 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.219464 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-kube-api-access-625wc\" (UniqueName: \"kubernetes.io/projected/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-kube-api-access-625wc\") pod \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " Oct 30 18:59:48.219709 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.219631 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-maas-api-tls\") pod \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " Oct 30 18:59:48.219775 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.219733 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-metrics-certs\") pod \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\" (UID: \"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1\") " Oct 30 18:59:48.219937 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.219912 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-openshift-service-ca" pod "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" (UID: "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:48.220576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.220542 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:48.222276 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.222246 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-maas-api-tls" pod "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" (UID: "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:48.222357 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.222346 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-metrics-certs" pod "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" (UID: "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:48.223087 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.223066 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-kube-api-access-625wc" pod "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" (UID: "0dd7a1dd-60de-4624-9d5a-5e36f75aedb1"). InnerVolumeSpecName "kube-api-access-625wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:48.308162 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.308132 4582 generic.go:356] "Generic (PLEG): container finished" podID="0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" containerID="281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f" exitCode=0 Oct 30 18:59:48.308332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.308165 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" Oct 30 18:59:48.308332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.308213 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" event={"ID":"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1","Type":"ContainerDied","Data":"281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f"} Oct 30 18:59:48.308332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.308265 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr" event={"ID":"0dd7a1dd-60de-4624-9d5a-5e36f75aedb1","Type":"ContainerDied","Data":"831b724b61620f349a46528693ce7fed468685541f68f2673f700cfd8b6de38d"} Oct 30 18:59:48.308332 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.308294 4582 scope.go:122] "RemoveContainer" containerID="281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f" Oct 30 18:59:48.321664 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.321636 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625wc\" (UniqueName: \"kubernetes.io/projected/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-kube-api-access-625wc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:48.321771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.321670 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:48.321771 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.321686 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 18:59:48.322272 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.322250 4582 scope.go:122] "RemoveContainer" containerID="281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f" Oct 30 18:59:48.322564 ip-10-0-133-124 kubenswrapper[4582]: E1030 18:59:48.322547 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f\": container with ID starting with 281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f not found: ID does not exist" containerID="281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f" Oct 30 18:59:48.322632 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.322577 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f"} err="failed to get container status \"281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f\": rpc error: code = NotFound desc = could not find container \"281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f\": container with ID starting with 281be1de2cc06a69b710f80c2b46417f3ad9b7b46abbb656bdde9daed6c0620f not found: ID does not exist" Oct 30 18:59:48.365795 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.365716 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr"] Oct 30 18:59:48.372051 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.372009 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c52ec7-5667874bc-qmgjr"] Oct 30 18:59:48.858760 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:48.858728 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.000576 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:49.000538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.101940 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:49.101903 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.302496 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:49.302463 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" path="/var/lib/kubelet/pods/0dd7a1dd-60de-4624-9d5a-5e36f75aedb1/volumes" Oct 30 18:59:49.691383 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:49.691345 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx"] Oct 30 18:59:49.722078 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:49.722043 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:50.960728 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:50.960688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:52.788242 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:52.788211 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:54.729904 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:54.729869 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:56.538534 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:56.538502 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:57.794784 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:57.794752 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s"] Oct 30 18:59:57.795333 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:57.795020 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" containerID="cri-o://cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581" gracePeriod=30 Oct 30 18:59:57.806508 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:57.806480 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm"] Oct 30 18:59:57.806700 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:57.806678 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" containerID="cri-o://2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea" gracePeriod=30 Oct 30 18:59:57.902972 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:57.902939 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6"] Oct 30 18:59:57.903153 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:57.903123 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" containerID="cri-o://683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90" gracePeriod=30 Oct 30 18:59:58.082649 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:58.082564 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx"] Oct 30 18:59:58.082871 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:58.082775 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" containerID="cri-o://af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139" gracePeriod=30 Oct 30 18:59:58.360973 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:58.360888 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:59.548469 ip-10-0-133-124 kubenswrapper[4582]: I1030 18:59:59.548439 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:00.099551 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:00.099521 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:02.279570 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:02.279538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:03.048858 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.048086 4582 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:00:03.048858 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.048154 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:00:03.051467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.051436 4582 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:00:03.051606 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.051500 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:00:03.119265 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.119240 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 19:00:03.150891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.150853 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-socket\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151061 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.150914 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istiod-ca-cert\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151125 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151071 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-token\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151181 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151132 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-podinfo\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151203 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-data\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151233 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-kube-api-access-6wvpm\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-kube-api-access-6wvpm\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151352 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151333 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-certs\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151502 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151481 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-envoy\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151574 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151563 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-credential-socket\") pod \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\" (UID: \"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f\") " Oct 30 19:00:03.151880 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151807 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-data" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.152013 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.151990 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istiod-ca-cert" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:03.152137 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152114 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-credential-socket" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.152224 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152134 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-certs" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.152224 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152180 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-socket" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.153060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152491 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.153060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152516 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.153060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152529 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.153060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152543 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.153060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.152556 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.154704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.154594 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-kube-api-access-6wvpm" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "kube-api-access-6wvpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.154704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.154680 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-podinfo" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:03.155080 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.155053 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-token" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.156863 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.156806 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-envoy" pod "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" (UID: "ca2e7559-dbb4-4e25-b7d5-4acb80518a7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.206256 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.205933 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 19:00:03.247307 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.247277 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 19:00:03.258765 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.258714 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-certs\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.258962 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.258848 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-data\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.258962 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.258891 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-token\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-token\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.258962 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.258932 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-podinfo\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.259184 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.258997 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-socket\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.259184 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.259050 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-credential-socket\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.259308 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.259184 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-envoy\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.259308 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.259222 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5030f1-3eae-4d64-9a64-52ee30032821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5030f1-3eae-4d64-9a64-52ee30032821-istiod-ca-cert\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.259427 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.259387 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-kube-api-access-fvssk\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-kube-api-access-fvssk\") pod \"9a5030f1-3eae-4d64-9a64-52ee30032821\" (UID: \"9a5030f1-3eae-4d64-9a64-52ee30032821\") " Oct 30 19:00:03.260526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.260479 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.260526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.260521 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.260740 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.260539 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvpm\" (UniqueName: \"kubernetes.io/projected/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-kube-api-access-6wvpm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.260740 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.260556 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.267427 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.267396 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-data" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.267649 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.267614 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-certs" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.268244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.268214 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-socket" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.268365 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.268325 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-kube-api-access-fvssk" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "kube-api-access-fvssk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.268512 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.268489 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5030f1-3eae-4d64-9a64-52ee30032821-istiod-ca-cert" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:03.270371 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.270341 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-credential-socket" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.278265 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.278240 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-podinfo" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:03.278971 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.278947 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-envoy" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.282485 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.282334 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-token" pod "9a5030f1-3eae-4d64-9a64-52ee30032821" (UID: "9a5030f1-3eae-4d64-9a64-52ee30032821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.361803 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.361765 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-socket\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.361993 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.361927 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-certs\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.361993 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.361972 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-data\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.362123 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362056 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-kube-api-access-spndf\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-kube-api-access-spndf\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.362123 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362087 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af00f9be-3c33-41fb-a383-0664ffa85e81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af00f9be-3c33-41fb-a383-0664ffa85e81-istiod-ca-cert\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.362123 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362099 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-socket" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.362323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362218 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-credential-socket\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.362323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362258 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-token\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-token\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.362323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362308 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-data" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.362503 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362323 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-podinfo\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.362503 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362351 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-envoy\") pod \"af00f9be-3c33-41fb-a383-0664ffa85e81\" (UID: \"af00f9be-3c33-41fb-a383-0664ffa85e81\") " Oct 30 19:00:03.362616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362529 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-certs" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.362718 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.362701 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-credential-socket" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363283 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363308 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363321 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363333 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363346 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363357 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363370 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363388 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363403 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363417 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5030f1-3eae-4d64-9a64-52ee30032821-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363429 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5030f1-3eae-4d64-9a64-52ee30032821-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363441 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.363453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.363456 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvssk\" (UniqueName: \"kubernetes.io/projected/9a5030f1-3eae-4d64-9a64-52ee30032821-kube-api-access-fvssk\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.365231 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.365208 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af00f9be-3c33-41fb-a383-0664ffa85e81-istiod-ca-cert" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:03.366642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.366619 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-kube-api-access-spndf" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "kube-api-access-spndf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.367799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.367737 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-podinfo" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:03.368153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.368129 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-envoy" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.368430 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.368391 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-token" pod "af00f9be-3c33-41fb-a383-0664ffa85e81" (UID: "af00f9be-3c33-41fb-a383-0664ffa85e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.424461 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.424435 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 19:00:03.439795 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.439764 4582 generic.go:356] "Generic (PLEG): container finished" podID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerID="683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90" exitCode=0 Oct 30 19:00:03.440000 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.439880 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" Oct 30 19:00:03.440073 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.439867 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" event={"ID":"af00f9be-3c33-41fb-a383-0664ffa85e81","Type":"ContainerDied","Data":"683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90"} Oct 30 19:00:03.440139 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.440074 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6" event={"ID":"af00f9be-3c33-41fb-a383-0664ffa85e81","Type":"ContainerDied","Data":"5f77e46f0ed10678f569758fa06a54d513cc0fdf55e71166c9c978301b37887e"} Oct 30 19:00:03.440139 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.440114 4582 scope.go:122] "RemoveContainer" containerID="683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90" Oct 30 19:00:03.453368 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.451854 4582 generic.go:356] "Generic (PLEG): container finished" podID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerID="2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea" exitCode=0 Oct 30 19:00:03.453368 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.452005 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" Oct 30 19:00:03.453368 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.452070 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" event={"ID":"9a5030f1-3eae-4d64-9a64-52ee30032821","Type":"ContainerDied","Data":"2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea"} Oct 30 19:00:03.453368 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.452113 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm" event={"ID":"9a5030f1-3eae-4d64-9a64-52ee30032821","Type":"ContainerDied","Data":"1e79e2d4f5ecfa33f02267a170d8ba6f5d3209e2cee020cc3fa0f885758fbe0e"} Oct 30 19:00:03.454199 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.454174 4582 generic.go:356] "Generic (PLEG): container finished" podID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerID="cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581" exitCode=0 Oct 30 19:00:03.454359 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.454340 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" event={"ID":"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f","Type":"ContainerDied","Data":"cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581"} Oct 30 19:00:03.454441 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.454390 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" event={"ID":"ca2e7559-dbb4-4e25-b7d5-4acb80518a7f","Type":"ContainerDied","Data":"6cc87de0bf56a25f1d11ef8edb55d22060dd73a6b1d7f1a5a62ad5340440dae0"} Oct 30 19:00:03.454441 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.454427 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s" Oct 30 19:00:03.456232 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.456206 4582 generic.go:356] "Generic (PLEG): container finished" podID="e6313580-767d-435f-a756-a2ffe1de64fa" containerID="af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139" exitCode=0 Oct 30 19:00:03.456342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.456291 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" event={"ID":"e6313580-767d-435f-a756-a2ffe1de64fa","Type":"ContainerDied","Data":"af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139"} Oct 30 19:00:03.456342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.456330 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" event={"ID":"e6313580-767d-435f-a756-a2ffe1de64fa","Type":"ContainerDied","Data":"8a21a73f9ac877cad0d8b16ab34ca9ed0282f25895e3349bd0db005b5ba77ed8"} Oct 30 19:00:03.456470 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.456396 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx" Oct 30 19:00:03.463947 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.463852 4582 scope.go:122] "RemoveContainer" containerID="683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90" Oct 30 19:00:03.464983 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.464950 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-envoy\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.465126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.465020 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-certs\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.465126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.465067 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-socket\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.465277 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.465145 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-kube-api-access-znb6t\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-kube-api-access-znb6t\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.465782 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:03.465753 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90\": container with ID starting with 683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90 not found: ID does not exist" containerID="683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90" Oct 30 19:00:03.465912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.465789 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90"} err="failed to get container status \"683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90\": rpc error: code = NotFound desc = could not find container \"683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90\": container with ID starting with 683e684796d1ad74b740adf37b19c046675c5165c732c96875a2fb30e5484d90 not found: ID does not exist" Oct 30 19:00:03.465912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.465812 4582 scope.go:122] "RemoveContainer" containerID="2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea" Oct 30 19:00:03.466238 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.466214 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-certs" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.468157 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.468131 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-socket" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.470805 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.470768 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-kube-api-access-znb6t" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "kube-api-access-znb6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.472086 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472067 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-data\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.472203 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472164 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6313580-767d-435f-a756-a2ffe1de64fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6313580-767d-435f-a756-a2ffe1de64fa-istio-podinfo\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.472297 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472284 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-istio-token\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-istio-token\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.472362 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472321 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6313580-767d-435f-a756-a2ffe1de64fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6313580-767d-435f-a756-a2ffe1de64fa-istiod-ca-cert\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.472424 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472381 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-credential-socket\") pod \"e6313580-767d-435f-a756-a2ffe1de64fa\" (UID: \"e6313580-767d-435f-a756-a2ffe1de64fa\") " Oct 30 19:00:03.472909 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472883 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znb6t\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-kube-api-access-znb6t\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.473002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472908 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spndf\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-kube-api-access-spndf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.473002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472923 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af00f9be-3c33-41fb-a383-0664ffa85e81-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.473002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472936 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.473002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472948 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.473002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472961 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af00f9be-3c33-41fb-a383-0664ffa85e81-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.473002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472975 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.473002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.472987 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.474180 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.473203 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-credential-socket" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.474180 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.473702 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-data" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.474180 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.474010 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6313580-767d-435f-a756-a2ffe1de64fa-istiod-ca-cert" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:03.476642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.476613 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-istio-token" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:03.477749 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.477718 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6313580-767d-435f-a756-a2ffe1de64fa-istio-podinfo" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:03.482144 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.482119 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-envoy" pod "e6313580-767d-435f-a756-a2ffe1de64fa" (UID: "e6313580-767d-435f-a756-a2ffe1de64fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:03.484877 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.484858 4582 scope.go:122] "RemoveContainer" containerID="2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea" Oct 30 19:00:03.485470 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:03.485211 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea\": container with ID starting with 2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea not found: ID does not exist" containerID="2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea" Oct 30 19:00:03.485470 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.485251 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea"} err="failed to get container status \"2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea\": rpc error: code = NotFound desc = could not find container \"2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea\": container with ID starting with 2c3ad556cbc34d0da40f434b4695ff2e46ead31ef6854adc4355769cab3f7bea not found: ID does not exist" Oct 30 19:00:03.485470 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.485273 4582 scope.go:122] "RemoveContainer" containerID="cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581" Oct 30 19:00:03.502852 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.502684 4582 scope.go:122] "RemoveContainer" containerID="cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581" Oct 30 19:00:03.503135 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:03.503084 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581\": container with ID starting with cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581 not found: ID does not exist" containerID="cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581" Oct 30 19:00:03.503135 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.503121 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581"} err="failed to get container status \"cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581\": rpc error: code = NotFound desc = could not find container \"cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581\": container with ID starting with cb79e5974a214f0ed479ebfb731489a4598cd67c47075bc7753f794cacb79581 not found: ID does not exist" Oct 30 19:00:03.503282 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.503145 4582 scope.go:122] "RemoveContainer" containerID="af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139" Oct 30 19:00:03.509459 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.509437 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6"] Oct 30 19:00:03.516068 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.516048 4582 scope.go:122] "RemoveContainer" containerID="af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139" Oct 30 19:00:03.516345 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:03.516310 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139\": container with ID starting with af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139 not found: ID does not exist" containerID="af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139" Oct 30 19:00:03.516434 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.516356 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139"} err="failed to get container status \"af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139\": rpc error: code = NotFound desc = could not find container \"af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139\": container with ID starting with af4422763d1577e5eea1788b8b4f6e03885c81823a2d3360b260e1832b31d139 not found: ID does not exist" Oct 30 19:00:03.519844 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.519797 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-bg9v6"] Oct 30 19:00:03.568214 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.568181 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s"] Oct 30 19:00:03.571907 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.571842 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-p6n4s"] Oct 30 19:00:03.574485 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.574463 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.574556 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.574488 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6313580-767d-435f-a756-a2ffe1de64fa-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.574556 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.574496 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6313580-767d-435f-a756-a2ffe1de64fa-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.574556 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.574508 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6313580-767d-435f-a756-a2ffe1de64fa-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.574556 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.574516 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.574556 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.574527 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6313580-767d-435f-a756-a2ffe1de64fa-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:03.603766 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.603735 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm"] Oct 30 19:00:03.611231 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.611202 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-5hmsm"] Oct 30 19:00:03.816004 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.815964 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx"] Oct 30 19:00:03.830723 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:03.830414 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-nqmgx"] Oct 30 19:00:04.682068 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:04.682035 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:04.713405 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:04.713362 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" podUID="d375b664-fdf7-4634-8a74-495d75055a1c" containerName="main" containerID="cri-o://1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26" gracePeriod=45 Oct 30 19:00:05.021935 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.021910 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 19:00:05.100099 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.100066 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d375b664-fdf7-4634-8a74-495d75055a1c-kube-api-access-vm2lf\" (UniqueName: \"kubernetes.io/projected/d375b664-fdf7-4634-8a74-495d75055a1c-kube-api-access-vm2lf\") pod \"d375b664-fdf7-4634-8a74-495d75055a1c\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " Oct 30 19:00:05.100375 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.100255 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-dshm\") pod \"d375b664-fdf7-4634-8a74-495d75055a1c\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " Oct 30 19:00:05.100811 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.100783 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-home" pod "d375b664-fdf7-4634-8a74-495d75055a1c" (UID: "d375b664-fdf7-4634-8a74-495d75055a1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:05.100969 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.100288 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-home\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-home\") pod \"d375b664-fdf7-4634-8a74-495d75055a1c\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " Oct 30 19:00:05.101090 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.100997 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d375b664-fdf7-4634-8a74-495d75055a1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d375b664-fdf7-4634-8a74-495d75055a1c-tls-certs\") pod \"d375b664-fdf7-4634-8a74-495d75055a1c\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " Oct 30 19:00:05.101440 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.101417 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-tmp-dir\") pod \"d375b664-fdf7-4634-8a74-495d75055a1c\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " Oct 30 19:00:05.101702 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.101679 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-tmp-dir" pod "d375b664-fdf7-4634-8a74-495d75055a1c" (UID: "d375b664-fdf7-4634-8a74-495d75055a1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:05.103288 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.103242 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-model-cache" pod "d375b664-fdf7-4634-8a74-495d75055a1c" (UID: "d375b664-fdf7-4634-8a74-495d75055a1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:05.103398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.103297 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d375b664-fdf7-4634-8a74-495d75055a1c-kube-api-access-vm2lf" pod "d375b664-fdf7-4634-8a74-495d75055a1c" (UID: "d375b664-fdf7-4634-8a74-495d75055a1c"). InnerVolumeSpecName "kube-api-access-vm2lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:05.103508 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.103490 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-model-cache\") pod \"d375b664-fdf7-4634-8a74-495d75055a1c\" (UID: \"d375b664-fdf7-4634-8a74-495d75055a1c\") " Oct 30 19:00:05.105075 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.105044 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:05.105075 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.105076 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:05.105252 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.105090 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm2lf\" (UniqueName: \"kubernetes.io/projected/d375b664-fdf7-4634-8a74-495d75055a1c-kube-api-access-vm2lf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:05.105252 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.105104 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:05.108232 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.108207 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-dshm" pod "d375b664-fdf7-4634-8a74-495d75055a1c" (UID: "d375b664-fdf7-4634-8a74-495d75055a1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:05.108344 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.108287 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d375b664-fdf7-4634-8a74-495d75055a1c-tls-certs" pod "d375b664-fdf7-4634-8a74-495d75055a1c" (UID: "d375b664-fdf7-4634-8a74-495d75055a1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:05.206196 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.206158 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d375b664-fdf7-4634-8a74-495d75055a1c-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:05.206196 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.206187 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d375b664-fdf7-4634-8a74-495d75055a1c-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:05.302450 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.302364 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" path="/var/lib/kubelet/pods/9a5030f1-3eae-4d64-9a64-52ee30032821/volumes" Oct 30 19:00:05.302982 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.302967 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" path="/var/lib/kubelet/pods/af00f9be-3c33-41fb-a383-0664ffa85e81/volumes" Oct 30 19:00:05.303481 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.303466 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" path="/var/lib/kubelet/pods/ca2e7559-dbb4-4e25-b7d5-4acb80518a7f/volumes" Oct 30 19:00:05.304328 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.304312 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" path="/var/lib/kubelet/pods/e6313580-767d-435f-a756-a2ffe1de64fa/volumes" Oct 30 19:00:05.476358 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.476327 4582 generic.go:356] "Generic (PLEG): container finished" podID="d375b664-fdf7-4634-8a74-495d75055a1c" containerID="1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26" exitCode=0 Oct 30 19:00:05.476538 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.476376 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" Oct 30 19:00:05.476538 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.476422 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" event={"ID":"d375b664-fdf7-4634-8a74-495d75055a1c","Type":"ContainerDied","Data":"1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26"} Oct 30 19:00:05.476538 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.476469 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx" event={"ID":"d375b664-fdf7-4634-8a74-495d75055a1c","Type":"ContainerDied","Data":"a4275b6d7167459045824118e350e0c74fd06d43068dffbc6377cde528816bed"} Oct 30 19:00:05.476538 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.476490 4582 scope.go:122] "RemoveContainer" containerID="1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26" Oct 30 19:00:05.486914 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.486892 4582 scope.go:122] "RemoveContainer" containerID="1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26" Oct 30 19:00:05.487232 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:05.487213 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26\": container with ID starting with 1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26 not found: ID does not exist" containerID="1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26" Oct 30 19:00:05.487304 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.487241 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26"} err="failed to get container status \"1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26\": rpc error: code = NotFound desc = could not find container \"1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26\": container with ID starting with 1013f2e6d237b07ec552ab3bcfae2f1ba0009f4b9ddc21e09616f50c6649ba26 not found: ID does not exist" Oct 30 19:00:05.530035 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.530003 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx"] Oct 30 19:00:05.536633 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:05.536608 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1aa59a/test-model-1aa59a-kserve-59b56cbf6c-8c2fx"] Oct 30 19:00:06.227059 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:06.227025 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:00:06.914372 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:06.914337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:07.165286 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:07.165206 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:07.285560 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:07.285527 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:07.302983 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:07.302955 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d375b664-fdf7-4634-8a74-495d75055a1c" path="/var/lib/kubelet/pods/d375b664-fdf7-4634-8a74-495d75055a1c/volumes" Oct 30 19:00:07.586173 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:07.586096 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:08.582681 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:08.582649 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:08.703579 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:08.703550 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:08.801721 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:08.801692 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:08.980131 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:08.980099 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:09.087560 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:09.087528 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:10.973866 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:10.973800 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.273371 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.273331 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.644126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644092 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6"] Oct 30 19:00:12.644897 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644883 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" Oct 30 19:00:12.644950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644899 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" Oct 30 19:00:12.644950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644914 4582 cpu_manager.go:414] "containerMap: removing container" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" Oct 30 19:00:12.644950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644920 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" Oct 30 19:00:12.644950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644934 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d375b664-fdf7-4634-8a74-495d75055a1c" containerName="main" Oct 30 19:00:12.644950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644939 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d375b664-fdf7-4634-8a74-495d75055a1c" containerName="main" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644959 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" containerName="maas-api" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644964 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" containerName="maas-api" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644979 4582 cpu_manager.go:414] "containerMap: removing container" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644983 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644990 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0cb1fe32-df10-442d-917f-090ee73cfb32" containerName="maas-api" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.644994 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb1fe32-df10-442d-917f-090ee73cfb32" containerName="maas-api" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645012 4582 cpu_manager.go:414] "containerMap: removing container" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" Oct 30 19:00:12.645117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645016 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" Oct 30 19:00:12.645408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645121 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5030f1-3eae-4d64-9a64-52ee30032821" containerName="istio-proxy" Oct 30 19:00:12.645408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645130 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="af00f9be-3c33-41fb-a383-0664ffa85e81" containerName="istio-proxy" Oct 30 19:00:12.645408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645138 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d375b664-fdf7-4634-8a74-495d75055a1c" containerName="main" Oct 30 19:00:12.645408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645149 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd7a1dd-60de-4624-9d5a-5e36f75aedb1" containerName="maas-api" Oct 30 19:00:12.645408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645157 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2e7559-dbb4-4e25-b7d5-4acb80518a7f" containerName="istio-proxy" Oct 30 19:00:12.645408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645163 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb1fe32-df10-442d-917f-090ee73cfb32" containerName="maas-api" Oct 30 19:00:12.645408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.645170 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6313580-767d-435f-a756-a2ffe1de64fa" containerName="istio-proxy" Oct 30 19:00:12.650429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.650409 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.665237 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.665215 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6"] Oct 30 19:00:12.672909 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.672880 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:12.672909 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.672889 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:00:12.715025 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.714992 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.715206 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.715117 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.715298 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.715260 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8zs\" (UniqueName: \"kubernetes.io/projected/5f371644-3e5c-4ce0-958f-921853b7d6d8-kube-api-access-6x8zs\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.816734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.816669 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.816734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.816740 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.817085 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.816929 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8zs\" (UniqueName: \"kubernetes.io/projected/5f371644-3e5c-4ce0-958f-921853b7d6d8-kube-api-access-6x8zs\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.817462 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.817439 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.819289 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.819270 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.836447 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.836420 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8zs\" (UniqueName: \"kubernetes.io/projected/5f371644-3e5c-4ce0-958f-921853b7d6d8-kube-api-access-6x8zs\") pod \"maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:12.966508 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:12.966444 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:13.188149 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:13.188113 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:13.271882 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:13.271847 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6"] Oct 30 19:00:13.272288 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:00:13.272264 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f371644_3e5c_4ce0_958f_921853b7d6d8.slice/crio-52a1fd2d44dd1edfc6b9f6285411f06bd2359f1a9d099a44224bac3b0d664359 WatchSource:0}: Error finding container 52a1fd2d44dd1edfc6b9f6285411f06bd2359f1a9d099a44224bac3b0d664359: Status 404 returned error can't find the container with id 52a1fd2d44dd1edfc6b9f6285411f06bd2359f1a9d099a44224bac3b0d664359 Oct 30 19:00:13.539548 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:13.539461 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" event={"ID":"5f371644-3e5c-4ce0-958f-921853b7d6d8","Type":"ContainerStarted","Data":"52a1fd2d44dd1edfc6b9f6285411f06bd2359f1a9d099a44224bac3b0d664359"} Oct 30 19:00:14.164710 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.164667 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf"] Oct 30 19:00:14.170739 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.170719 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.187137 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.187088 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf"] Oct 30 19:00:14.194232 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.194205 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-869b55\"/\"openshift-service-ca.crt\"" Oct 30 19:00:14.194377 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.194239 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-869b55\"/\"kube-root-ca.crt\"" Oct 30 19:00:14.194377 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.194266 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-869b55\"/\"test-model-869b55-kserve-self-signed-certs\"" Oct 30 19:00:14.198235 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.198216 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-869b55\"/\"default-dockercfg-zrp5x\"" Oct 30 19:00:14.211439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.211407 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:14.236755 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.236728 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-model-cache\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.236951 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.236816 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-dshm\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.236951 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.236920 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-tmp-dir\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.237106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.236970 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-home\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.237170 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.237104 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b78c648d-750c-4480-8cec-ab2aa12cc480-tls-certs\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.237233 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.237207 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvxx\" (UniqueName: \"kubernetes.io/projected/b78c648d-750c-4480-8cec-ab2aa12cc480-kube-api-access-qgvxx\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.338614 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.338576 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvxx\" (UniqueName: \"kubernetes.io/projected/b78c648d-750c-4480-8cec-ab2aa12cc480-kube-api-access-qgvxx\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.338809 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.338625 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-model-cache\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.338809 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.338732 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-dshm\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.338809 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.338767 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-tmp-dir\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.339058 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.338815 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-home\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.339137 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.339118 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b78c648d-750c-4480-8cec-ab2aa12cc480-tls-certs\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.339208 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.339147 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-model-cache\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.339208 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.339191 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-tmp-dir\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.339455 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.339432 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-home\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.341029 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.341009 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-dshm\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.341509 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.341491 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b78c648d-750c-4480-8cec-ab2aa12cc480-tls-certs\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.357965 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.357934 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvxx\" (UniqueName: \"kubernetes.io/projected/b78c648d-750c-4480-8cec-ab2aa12cc480-kube-api-access-qgvxx\") pod \"test-model-869b55-kserve-76b6d97df-zg8tf\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.488861 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.488752 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:14.550908 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.550872 4582 generic.go:356] "Generic (PLEG): container finished" podID="5f371644-3e5c-4ce0-958f-921853b7d6d8" containerID="363be81559d3cbefd8a5384127889d8dc7385d1e862ae855900f99934ad3d75a" exitCode=0 Oct 30 19:00:14.551329 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.551028 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" event={"ID":"5f371644-3e5c-4ce0-958f-921853b7d6d8","Type":"ContainerDied","Data":"363be81559d3cbefd8a5384127889d8dc7385d1e862ae855900f99934ad3d75a"} Oct 30 19:00:14.667760 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.667728 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:14.798921 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:14.798896 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf"] Oct 30 19:00:14.799951 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:00:14.799926 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb78c648d_750c_4480_8cec_ab2aa12cc480.slice/crio-9cc344343ef4108c36a7f7515f157d22350020a337a2f4800c888b9143c5cf57 WatchSource:0}: Error finding container 9cc344343ef4108c36a7f7515f157d22350020a337a2f4800c888b9143c5cf57: Status 404 returned error can't find the container with id 9cc344343ef4108c36a7f7515f157d22350020a337a2f4800c888b9143c5cf57 Oct 30 19:00:15.266151 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.266116 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:15.560359 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.560261 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" event={"ID":"b78c648d-750c-4480-8cec-ab2aa12cc480","Type":"ContainerStarted","Data":"4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8"} Oct 30 19:00:15.560359 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.560308 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" event={"ID":"b78c648d-750c-4480-8cec-ab2aa12cc480","Type":"ContainerStarted","Data":"9cc344343ef4108c36a7f7515f157d22350020a337a2f4800c888b9143c5cf57"} Oct 30 19:00:15.604465 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.604398 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" podStartSLOduration=1.6043786820000001 podStartE2EDuration="1.604378682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:15.599921002 +0000 UTC m=+3712.424315296" watchObservedRunningTime="2026-10-30 19:00:15.604378682 +0000 UTC m=+3712.428772976" Oct 30 19:00:15.733065 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.733037 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:15.759128 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.759095 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f371644-3e5c-4ce0-958f-921853b7d6d8-kube-api-access-6x8zs\" (UniqueName: \"kubernetes.io/projected/5f371644-3e5c-4ce0-958f-921853b7d6d8-kube-api-access-6x8zs\") pod \"5f371644-3e5c-4ce0-958f-921853b7d6d8\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " Oct 30 19:00:15.759342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.759328 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-xks-ca\") pod \"5f371644-3e5c-4ce0-958f-921853b7d6d8\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " Oct 30 19:00:15.759416 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.759404 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-openshift-ca\") pod \"5f371644-3e5c-4ce0-958f-921853b7d6d8\" (UID: \"5f371644-3e5c-4ce0-958f-921853b7d6d8\") " Oct 30 19:00:15.759795 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.759769 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-openshift-ca" pod "5f371644-3e5c-4ce0-958f-921853b7d6d8" (UID: "5f371644-3e5c-4ce0-958f-921853b7d6d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:15.761768 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.761740 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-xks-ca" pod "5f371644-3e5c-4ce0-958f-921853b7d6d8" (UID: "5f371644-3e5c-4ce0-958f-921853b7d6d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:15.761882 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.761784 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f371644-3e5c-4ce0-958f-921853b7d6d8-kube-api-access-6x8zs" pod "5f371644-3e5c-4ce0-958f-921853b7d6d8" (UID: "5f371644-3e5c-4ce0-958f-921853b7d6d8"). InnerVolumeSpecName "kube-api-access-6x8zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:15.861589 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.861498 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8zs\" (UniqueName: \"kubernetes.io/projected/5f371644-3e5c-4ce0-958f-921853b7d6d8-kube-api-access-6x8zs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:15.861589 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.861535 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:15.861589 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:15.861548 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f371644-3e5c-4ce0-958f-921853b7d6d8-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:16.489224 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:16.489195 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:16.568696 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:16.568662 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" Oct 30 19:00:16.568696 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:16.568686 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6" event={"ID":"5f371644-3e5c-4ce0-958f-921853b7d6d8","Type":"ContainerDied","Data":"52a1fd2d44dd1edfc6b9f6285411f06bd2359f1a9d099a44224bac3b0d664359"} Oct 30 19:00:16.569154 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:16.568735 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a1fd2d44dd1edfc6b9f6285411f06bd2359f1a9d099a44224bac3b0d664359" Oct 30 19:00:17.110589 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:17.110553 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:18.636514 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:18.636437 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:18.976219 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:18.976129 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:21.014916 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:21.014881 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:22.881444 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:22.881403 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:23.310013 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:23.309908 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:23.416968 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:23.416934 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:24.489636 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:24.489600 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:24.502357 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:24.502328 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:24.640051 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:24.640018 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:00:24.971188 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:24.971151 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:26.765933 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:26.765897 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:29.135565 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:29.135527 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:31.127050 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:31.127008 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:32.870926 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:32.870888 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:34.578055 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:34.578019 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:34.666123 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:34.666085 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:36.228156 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:36.228122 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:00:36.567120 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:36.567041 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:38.325237 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:38.325179 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:38.391150 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:38.391118 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m"] Oct 30 19:00:38.432496 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:38.432463 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:40.268666 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:40.268622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:41.667416 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:41.667374 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:42.175367 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:42.175333 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:43.865176 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:43.865089 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:45.572499 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:45.572448 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:47.378850 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:47.378797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:49.159119 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:49.159086 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:50.915911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:50.915879 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:52.696791 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:52.696753 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:53.412879 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.412814 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" podUID="f5581b97-02bd-4aa6-b85c-c1ef2813795f" containerName="main" containerID="cri-o://f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114" gracePeriod=45 Oct 30 19:00:53.695581 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.695554 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 19:00:53.779209 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779179 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-home\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-home\") pod \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779256 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-model-cache\") pod \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779400 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tmp-dir\") pod \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779448 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5581b97-02bd-4aa6-b85c-c1ef2813795f-kube-api-access-pl9rp\" (UniqueName: \"kubernetes.io/projected/f5581b97-02bd-4aa6-b85c-c1ef2813795f-kube-api-access-pl9rp\") pod \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779480 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-dshm\") pod \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779504 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-model-cache" pod "f5581b97-02bd-4aa6-b85c-c1ef2813795f" (UID: "f5581b97-02bd-4aa6-b85c-c1ef2813795f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779514 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-home" pod "f5581b97-02bd-4aa6-b85c-c1ef2813795f" (UID: "f5581b97-02bd-4aa6-b85c-c1ef2813795f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779537 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tls-certs\") pod \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\" (UID: \"f5581b97-02bd-4aa6-b85c-c1ef2813795f\") " Oct 30 19:00:53.779704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.779672 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tmp-dir" pod "f5581b97-02bd-4aa6-b85c-c1ef2813795f" (UID: "f5581b97-02bd-4aa6-b85c-c1ef2813795f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:53.780230 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.780210 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:53.780289 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.780237 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:53.780289 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.780251 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:53.781779 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.781749 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-dshm" pod "f5581b97-02bd-4aa6-b85c-c1ef2813795f" (UID: "f5581b97-02bd-4aa6-b85c-c1ef2813795f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:53.781907 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.781789 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tls-certs" pod "f5581b97-02bd-4aa6-b85c-c1ef2813795f" (UID: "f5581b97-02bd-4aa6-b85c-c1ef2813795f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:53.781907 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.781851 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5581b97-02bd-4aa6-b85c-c1ef2813795f-kube-api-access-pl9rp" pod "f5581b97-02bd-4aa6-b85c-c1ef2813795f" (UID: "f5581b97-02bd-4aa6-b85c-c1ef2813795f"). InnerVolumeSpecName "kube-api-access-pl9rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:53.848239 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.848205 4582 generic.go:356] "Generic (PLEG): container finished" podID="f5581b97-02bd-4aa6-b85c-c1ef2813795f" containerID="f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114" exitCode=0 Oct 30 19:00:53.848429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.848260 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" Oct 30 19:00:53.848429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.848287 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" event={"ID":"f5581b97-02bd-4aa6-b85c-c1ef2813795f","Type":"ContainerDied","Data":"f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114"} Oct 30 19:00:53.848429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.848339 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m" event={"ID":"f5581b97-02bd-4aa6-b85c-c1ef2813795f","Type":"ContainerDied","Data":"4c81d114caeac1fbb88874b9108c739c3404cc8bd8c7e5fd3c153649b767dcd5"} Oct 30 19:00:53.848429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.848362 4582 scope.go:122] "RemoveContainer" containerID="f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114" Oct 30 19:00:53.859280 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.859265 4582 scope.go:122] "RemoveContainer" containerID="f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114" Oct 30 19:00:53.859532 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:00:53.859511 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114\": container with ID starting with f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114 not found: ID does not exist" containerID="f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114" Oct 30 19:00:53.859584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.859539 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114"} err="failed to get container status \"f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114\": rpc error: code = NotFound desc = could not find container \"f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114\": container with ID starting with f02c3aab4a15e65c82656f9ce760ef438f7b76b108073cdc00d6da7e0c3a6114 not found: ID does not exist" Oct 30 19:00:53.881163 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.881138 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl9rp\" (UniqueName: \"kubernetes.io/projected/f5581b97-02bd-4aa6-b85c-c1ef2813795f-kube-api-access-pl9rp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:53.881163 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.881160 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5581b97-02bd-4aa6-b85c-c1ef2813795f-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:53.881342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.881169 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5581b97-02bd-4aa6-b85c-c1ef2813795f-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:53.906053 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.906027 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m"] Oct 30 19:00:53.909765 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:53.909743 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8862aa/test-model-8862aa-kserve-59d8b5c5-75c7m"] Oct 30 19:00:54.474888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:54.474855 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:55.301165 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:55.301130 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5581b97-02bd-4aa6-b85c-c1ef2813795f" path="/var/lib/kubelet/pods/f5581b97-02bd-4aa6-b85c-c1ef2813795f/volumes" Oct 30 19:00:56.221313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:56.221281 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:57.729630 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:57.729596 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw"] Oct 30 19:00:57.730103 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:57.729881 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" podUID="f24a3e46-8357-43b8-ab2c-9f41aaa929a4" containerName="maas-api" containerID="cri-o://2a76951b264549d22584dc208b6bf64066ca5ac3dd7ee7299cc410a2a19f4de4" gracePeriod=30 Oct 30 19:00:57.815645 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:57.815610 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:57.879324 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:57.879290 4582 generic.go:356] "Generic (PLEG): container finished" podID="f24a3e46-8357-43b8-ab2c-9f41aaa929a4" containerID="2a76951b264549d22584dc208b6bf64066ca5ac3dd7ee7299cc410a2a19f4de4" exitCode=0 Oct 30 19:00:57.879493 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:57.879345 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" event={"ID":"f24a3e46-8357-43b8-ab2c-9f41aaa929a4","Type":"ContainerDied","Data":"2a76951b264549d22584dc208b6bf64066ca5ac3dd7ee7299cc410a2a19f4de4"} Oct 30 19:00:58.000705 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.000623 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 19:00:58.034502 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.034465 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-maas-api-tls\") pod \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " Oct 30 19:00:58.034707 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.034682 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-openshift-service-ca\") pod \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " Oct 30 19:00:58.034787 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.034728 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-metrics-certs\") pod \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " Oct 30 19:00:58.034888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.034795 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-kube-api-access-dspdd\" (UniqueName: \"kubernetes.io/projected/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-kube-api-access-dspdd\") pod \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\" (UID: \"f24a3e46-8357-43b8-ab2c-9f41aaa929a4\") " Oct 30 19:00:58.035135 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.035110 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-openshift-service-ca" pod "f24a3e46-8357-43b8-ab2c-9f41aaa929a4" (UID: "f24a3e46-8357-43b8-ab2c-9f41aaa929a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:58.035514 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.035491 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:58.037002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.036976 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-maas-api-tls" pod "f24a3e46-8357-43b8-ab2c-9f41aaa929a4" (UID: "f24a3e46-8357-43b8-ab2c-9f41aaa929a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:58.037094 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.037045 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-kube-api-access-dspdd" pod "f24a3e46-8357-43b8-ab2c-9f41aaa929a4" (UID: "f24a3e46-8357-43b8-ab2c-9f41aaa929a4"). InnerVolumeSpecName "kube-api-access-dspdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:58.037094 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.037064 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-metrics-certs" pod "f24a3e46-8357-43b8-ab2c-9f41aaa929a4" (UID: "f24a3e46-8357-43b8-ab2c-9f41aaa929a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:58.136616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.136582 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:58.136616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.136611 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dspdd\" (UniqueName: \"kubernetes.io/projected/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-kube-api-access-dspdd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:58.136616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.136622 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f24a3e46-8357-43b8-ab2c-9f41aaa929a4-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:00:58.636418 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.636384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:58.779289 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.779254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:58.856038 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.856004 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6"] Oct 30 19:00:58.875092 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.875060 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1aa59a-xgzj6"] Oct 30 19:00:58.888292 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.888229 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" Oct 30 19:00:58.888292 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.888261 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw" event={"ID":"f24a3e46-8357-43b8-ab2c-9f41aaa929a4","Type":"ContainerDied","Data":"1288ea798d05347e721582146bbe818cb86f62f08baac1282940547efb116b68"} Oct 30 19:00:58.888496 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.888314 4582 scope.go:122] "RemoveContainer" containerID="2a76951b264549d22584dc208b6bf64066ca5ac3dd7ee7299cc410a2a19f4de4" Oct 30 19:00:58.952118 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.952088 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw"] Oct 30 19:00:58.981390 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:58.981352 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1aa59a-57f85bd77-l6jjw"] Oct 30 19:00:59.065341 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:59.065311 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:59.301952 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:59.301880 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f371644-3e5c-4ce0-958f-921853b7d6d8" path="/var/lib/kubelet/pods/5f371644-3e5c-4ce0-958f-921853b7d6d8/volumes" Oct 30 19:00:59.302364 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:59.302350 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24a3e46-8357-43b8-ab2c-9f41aaa929a4" path="/var/lib/kubelet/pods/f24a3e46-8357-43b8-ab2c-9f41aaa929a4/volumes" Oct 30 19:00:59.336601 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:00:59.336569 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:00.850229 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:00.850189 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:00.963460 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:00.963426 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:03.120565 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:03.120527 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:04.985180 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:04.985149 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:06.227175 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:01:06.227141 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:01:06.820586 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:06.820532 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:06.968248 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:06.968211 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:07.153753 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:07.153722 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:09.811229 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:09.811195 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:12.193609 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:12.193568 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:12.728380 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:12.728345 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:13.195964 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:13.195914 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:16.846908 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:16.846871 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:18.006172 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:18.006135 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:20.345164 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:20.345126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:20.919343 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:20.919313 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:23.883640 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:23.883596 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:26.357058 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:26.357024 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:28.443882 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.443805 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j"] Oct 30 19:01:28.444954 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.444927 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f5581b97-02bd-4aa6-b85c-c1ef2813795f" containerName="main" Oct 30 19:01:28.444954 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.444951 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5581b97-02bd-4aa6-b85c-c1ef2813795f" containerName="main" Oct 30 19:01:28.445138 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.444971 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5f371644-3e5c-4ce0-958f-921853b7d6d8" containerName="revoke-keys" Oct 30 19:01:28.445138 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.444977 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f371644-3e5c-4ce0-958f-921853b7d6d8" containerName="revoke-keys" Oct 30 19:01:28.445138 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.445025 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f24a3e46-8357-43b8-ab2c-9f41aaa929a4" containerName="maas-api" Oct 30 19:01:28.445138 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.445030 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24a3e46-8357-43b8-ab2c-9f41aaa929a4" containerName="maas-api" Oct 30 19:01:28.445270 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.445169 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24a3e46-8357-43b8-ab2c-9f41aaa929a4" containerName="maas-api" Oct 30 19:01:28.445270 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.445179 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5581b97-02bd-4aa6-b85c-c1ef2813795f" containerName="main" Oct 30 19:01:28.445270 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.445188 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f371644-3e5c-4ce0-958f-921853b7d6d8" containerName="revoke-keys" Oct 30 19:01:28.449200 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.449179 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.556962 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.556923 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557147 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.556971 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgm5p\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-kube-api-access-dgm5p\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557147 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.557055 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557147 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.557110 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557278 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.557165 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d167647-e7ba-4ded-845e-e0737006aaab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557278 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.557190 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d167647-e7ba-4ded-845e-e0737006aaab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557356 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.557292 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557438 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.557419 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.557493 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.557481 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.631948 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.631920 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:28.652096 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.652065 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j"] Oct 30 19:01:28.659257 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659231 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659422 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659282 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659422 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659354 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659422 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659387 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgm5p\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-kube-api-access-dgm5p\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659619 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659438 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659619 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659462 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659619 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659518 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d167647-e7ba-4ded-845e-e0737006aaab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659619 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659547 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d167647-e7ba-4ded-845e-e0737006aaab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.659619 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659599 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.660000 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.659716 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.660070 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.660031 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.660070 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.660055 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.660313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.660288 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.661021 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.660996 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d167647-e7ba-4ded-845e-e0737006aaab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.662232 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.662205 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.662788 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.662762 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d167647-e7ba-4ded-845e-e0737006aaab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.684040 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.684009 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.689662 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.689635 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgm5p\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-kube-api-access-dgm5p\") pod \"maas-default-gateway-openshift-default-76597d6855-76h9j\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:28.767012 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:28.766941 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:29.154321 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:29.154294 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j"] Oct 30 19:01:29.155323 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:01:29.155300 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d167647_e7ba_4ded_845e_e0737006aaab.slice/crio-c4a1cd7924745c4df0401f88a619b634afe6129dc50ff74bd274fa8423b7b0f5 WatchSource:0}: Error finding container c4a1cd7924745c4df0401f88a619b634afe6129dc50ff74bd274fa8423b7b0f5: Status 404 returned error can't find the container with id c4a1cd7924745c4df0401f88a619b634afe6129dc50ff74bd274fa8423b7b0f5 Oct 30 19:01:29.158241 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:29.158210 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:01:29.158311 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:29.158276 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:01:29.158311 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:29.158307 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:01:29.601167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:29.601084 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:30.121803 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:30.121758 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" event={"ID":"3d167647-e7ba-4ded-845e-e0737006aaab","Type":"ContainerStarted","Data":"aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff"} Oct 30 19:01:30.122039 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:30.121895 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" event={"ID":"3d167647-e7ba-4ded-845e-e0737006aaab","Type":"ContainerStarted","Data":"c4a1cd7924745c4df0401f88a619b634afe6129dc50ff74bd274fa8423b7b0f5"} Oct 30 19:01:30.368701 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:30.368627 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podStartSLOduration=2.368602848 podStartE2EDuration="2.368602848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:30.362208493 +0000 UTC m=+3787.186602788" watchObservedRunningTime="2026-10-30 19:01:30.368602848 +0000 UTC m=+3787.192997143" Oct 30 19:01:30.460901 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:30.459679 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:30.767602 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:30.767508 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:30.770085 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:30.770054 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-76h9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 19:01:30.770242 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:30.770114 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 19:01:31.769276 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:31.769175 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-76h9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 19:01:31.769276 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:31.769242 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 19:01:31.935941 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:31.935904 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:32.767314 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:32.767279 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-76h9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 19:01:32.767530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:32.767333 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 19:01:33.768175 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:33.768139 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-76h9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 19:01:33.768794 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:33.768763 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 19:01:34.302258 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:34.302224 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:34.767447 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:34.767411 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-76h9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 19:01:34.767752 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:34.767720 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 19:01:35.767938 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:35.767903 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-76h9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 19:01:35.768419 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:35.767955 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 19:01:36.230338 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:01:36.230293 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:01:36.521480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:36.521386 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:36.767905 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:36.767872 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-76h9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 19:01:36.768091 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:36.767926 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 19:01:37.773171 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:37.773139 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:37.773568 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:37.773499 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:37.774143 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:37.774120 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:01:38.491465 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:38.491433 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:40.086299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.086255 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2"] Oct 30 19:01:40.086745 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.086495 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" podUID="5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" containerName="maas-api" containerID="cri-o://2da031c1a3008fbf0c7e8c50654458c724f07ae05d74bca2c32964dddbe5a0a8" gracePeriod=30 Oct 30 19:01:40.159330 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.159298 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:40.227375 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.227344 4582 generic.go:356] "Generic (PLEG): container finished" podID="5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" containerID="2da031c1a3008fbf0c7e8c50654458c724f07ae05d74bca2c32964dddbe5a0a8" exitCode=0 Oct 30 19:01:40.227543 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.227418 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" event={"ID":"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6","Type":"ContainerDied","Data":"2da031c1a3008fbf0c7e8c50654458c724f07ae05d74bca2c32964dddbe5a0a8"} Oct 30 19:01:40.363958 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.363896 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 19:01:40.449296 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.449262 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-openshift-service-ca\") pod \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " Oct 30 19:01:40.449530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.449318 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-maas-api-tls\") pod \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " Oct 30 19:01:40.449530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.449347 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-metrics-certs\") pod \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " Oct 30 19:01:40.449530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.449386 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-kube-api-access-57pvc\" (UniqueName: \"kubernetes.io/projected/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-kube-api-access-57pvc\") pod \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\" (UID: \"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6\") " Oct 30 19:01:40.450217 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.450176 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-openshift-service-ca" pod "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" (UID: "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:40.452269 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.452240 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-kube-api-access-57pvc" pod "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" (UID: "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6"). InnerVolumeSpecName "kube-api-access-57pvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:40.452629 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.452611 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-metrics-certs" pod "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" (UID: "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:40.452932 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.452907 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-maas-api-tls" pod "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" (UID: "5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:40.551522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.551491 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:40.551522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.551520 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:40.551687 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.551530 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:40.551687 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:40.551538 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pvc\" (UniqueName: \"kubernetes.io/projected/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6-kube-api-access-57pvc\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:41.016176 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.016141 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:41.127307 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.127245 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:41.219537 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.219505 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:41.236043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.236019 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" Oct 30 19:01:41.236225 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.236046 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2" event={"ID":"5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6","Type":"ContainerDied","Data":"00c198a92423780b930031002978938b82741ff38468c9a95ce7d052c6cd187f"} Oct 30 19:01:41.236225 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.236101 4582 scope.go:122] "RemoveContainer" containerID="2da031c1a3008fbf0c7e8c50654458c724f07ae05d74bca2c32964dddbe5a0a8" Oct 30 19:01:41.289628 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.289553 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2"] Oct 30 19:01:41.302952 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.302925 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8862aa-6bccf4f685-hzrl2"] Oct 30 19:01:41.845830 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:41.845763 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:43.302106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.302073 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" path="/var/lib/kubelet/pods/5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6/volumes" Oct 30 19:01:43.428168 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.428137 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7"] Oct 30 19:01:43.428918 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.428903 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" containerName="maas-api" Oct 30 19:01:43.428974 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.428919 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" containerName="maas-api" Oct 30 19:01:43.429081 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.429072 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5456574e-2ce7-4ab4-a0fa-e4e3ba0273f6" containerName="maas-api" Oct 30 19:01:43.432805 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.432786 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.457043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.457016 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7"] Oct 30 19:01:43.489639 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.489606 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.489877 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.489660 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.489877 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.489794 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.490024 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.489899 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70547fc1-ceea-47fd-abb0-c740d151c84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.490024 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.490006 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrwt\" (UniqueName: \"kubernetes.io/projected/70547fc1-ceea-47fd-abb0-c740d151c84c-kube-api-access-vmrwt\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.490145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.490046 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.490145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.490073 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.490260 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.490236 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.490327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.490282 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.526920 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.526893 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:43.591517 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591438 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.591517 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591492 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.591727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591529 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.591727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591560 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70547fc1-ceea-47fd-abb0-c740d151c84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.591727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591620 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrwt\" (UniqueName: \"kubernetes.io/projected/70547fc1-ceea-47fd-abb0-c740d151c84c-kube-api-access-vmrwt\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.591727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591645 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.591727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591661 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.591727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591709 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.592077 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.591750 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.592077 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.592009 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.592077 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.592055 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.592215 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.592154 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.592215 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.592186 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.592567 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.592551 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70547fc1-ceea-47fd-abb0-c740d151c84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.594077 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.594053 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.594299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.594281 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.615950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.615918 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrwt\" (UniqueName: \"kubernetes.io/projected/70547fc1-ceea-47fd-abb0-c740d151c84c-kube-api-access-vmrwt\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.616309 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.616291 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70547fc1-ceea-47fd-abb0-c740d151c84c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-tpnm7\" (UID: \"70547fc1-ceea-47fd-abb0-c740d151c84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:43.747499 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:43.747470 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:44.088726 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.088699 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7"] Oct 30 19:01:44.090389 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:01:44.090361 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70547fc1_ceea_47fd_abb0_c740d151c84c.slice/crio-a8e5f2c3dfee4c602b5d4e147c06a0fec7dfdbb2b61945f5d95d92a325c95e2d WatchSource:0}: Error finding container a8e5f2c3dfee4c602b5d4e147c06a0fec7dfdbb2b61945f5d95d92a325c95e2d: Status 404 returned error can't find the container with id a8e5f2c3dfee4c602b5d4e147c06a0fec7dfdbb2b61945f5d95d92a325c95e2d Oct 30 19:01:44.093404 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.093373 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:01:44.093469 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.093443 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:01:44.093519 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.093471 4582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:01:44.260727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.260692 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" event={"ID":"70547fc1-ceea-47fd-abb0-c740d151c84c","Type":"ContainerStarted","Data":"3640199b9be91dbde92bd2510a1e0c35124193820f840c23bcaa97b175f89746"} Oct 30 19:01:44.260919 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.260749 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" event={"ID":"70547fc1-ceea-47fd-abb0-c740d151c84c","Type":"ContainerStarted","Data":"a8e5f2c3dfee4c602b5d4e147c06a0fec7dfdbb2b61945f5d95d92a325c95e2d"} Oct 30 19:01:44.319311 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.319259 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podStartSLOduration=1.319245631 podStartE2EDuration="1.319245631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:44.316943776 +0000 UTC m=+3801.141338082" watchObservedRunningTime="2026-10-30 19:01:44.319245631 +0000 UTC m=+3801.143639922" Oct 30 19:01:44.747584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:44.747552 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:45.512977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:45.512935 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:45.761845 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:45.759919 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tpnm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 19:01:45.761845 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:45.760005 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podUID="70547fc1-ceea-47fd-abb0-c740d151c84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 19:01:46.762843 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:46.762160 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tpnm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 19:01:46.762843 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:46.762222 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podUID="70547fc1-ceea-47fd-abb0-c740d151c84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 19:01:47.567233 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:47.567197 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:47.750206 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:47.750166 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tpnm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 19:01:47.750484 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:47.750457 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podUID="70547fc1-ceea-47fd-abb0-c740d151c84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 19:01:48.757971 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:48.757884 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tpnm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 19:01:48.758495 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:48.758470 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podUID="70547fc1-ceea-47fd-abb0-c740d151c84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 19:01:48.806935 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:48.806899 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:48.991536 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:48.991495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:49.295362 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:49.293371 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5"] Oct 30 19:01:49.295362 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:49.293591 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" containerID="cri-o://31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a" gracePeriod=30 Oct 30 19:01:49.328345 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:49.328313 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:49.753854 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:49.753218 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tpnm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 19:01:49.753854 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:49.753296 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podUID="70547fc1-ceea-47fd-abb0-c740d151c84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 19:01:49.757838 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:49.754953 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:50.748317 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:50.748279 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tpnm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 19:01:50.748738 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:50.748357 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podUID="70547fc1-ceea-47fd-abb0-c740d151c84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 19:01:51.748097 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:51.748063 4582 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tpnm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 19:01:51.748280 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:51.748119 4582 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" podUID="70547fc1-ceea-47fd-abb0-c740d151c84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 19:01:51.802883 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:51.802845 4582 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:01:51.803266 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:51.802907 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:01:52.146993 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:52.146959 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:52.891096 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:52.891063 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:52.891596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:52.891546 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:52.892488 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:52.892466 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tpnm7" Oct 30 19:01:54.174283 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.174234 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:54.580556 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.580525 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 19:01:54.667769 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.667733 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-data\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.667780 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-certs\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.667855 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-istio-token\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-istio-token\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.667896 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-kube-api-access-ctxtv\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-kube-api-access-ctxtv\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.667937 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bed21b2d-f889-45c4-8d56-e5da10476136-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed21b2d-f889-45c4-8d56-e5da10476136-istio-podinfo\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.667995 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-socket\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668027 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-credential-socket\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668475 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668102 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-data" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.668475 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668132 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed21b2d-f889-45c4-8d56-e5da10476136-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed21b2d-f889-45c4-8d56-e5da10476136-istiod-ca-cert\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668475 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668139 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-certs" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.668475 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668193 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-envoy\") pod \"bed21b2d-f889-45c4-8d56-e5da10476136\" (UID: \"bed21b2d-f889-45c4-8d56-e5da10476136\") " Oct 30 19:01:54.668475 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668299 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-socket" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.668475 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668415 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-credential-socket" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.668740 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.668541 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed21b2d-f889-45c4-8d56-e5da10476136-istiod-ca-cert" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:54.669109 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.669080 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.669244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.669113 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.669244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.669129 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.669244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.669154 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.669244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.669171 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed21b2d-f889-45c4-8d56-e5da10476136-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.670301 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.670276 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bed21b2d-f889-45c4-8d56-e5da10476136-istio-podinfo" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:01:54.670623 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.670598 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-istio-token" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:54.670843 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.670803 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-envoy" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:54.670948 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.670859 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-kube-api-access-ctxtv" pod "bed21b2d-f889-45c4-8d56-e5da10476136" (UID: "bed21b2d-f889-45c4-8d56-e5da10476136"). InnerVolumeSpecName "kube-api-access-ctxtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:54.770578 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.770505 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed21b2d-f889-45c4-8d56-e5da10476136-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.770578 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.770530 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.770578 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.770541 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxtv\" (UniqueName: \"kubernetes.io/projected/bed21b2d-f889-45c4-8d56-e5da10476136-kube-api-access-ctxtv\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.770578 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:54.770552 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed21b2d-f889-45c4-8d56-e5da10476136-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:01:55.352305 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.352264 4582 generic.go:356] "Generic (PLEG): container finished" podID="bed21b2d-f889-45c4-8d56-e5da10476136" containerID="31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a" exitCode=0 Oct 30 19:01:55.352697 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.352329 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" Oct 30 19:01:55.352697 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.352336 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" event={"ID":"bed21b2d-f889-45c4-8d56-e5da10476136","Type":"ContainerDied","Data":"31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a"} Oct 30 19:01:55.352697 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.352391 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5" event={"ID":"bed21b2d-f889-45c4-8d56-e5da10476136","Type":"ContainerDied","Data":"886524aaaab745790868b2238f6e7a5c8a56f60d4047f6affc14cbc7d38b1638"} Oct 30 19:01:55.352697 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.352416 4582 scope.go:122] "RemoveContainer" containerID="31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a" Oct 30 19:01:55.363250 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.363225 4582 scope.go:122] "RemoveContainer" containerID="31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a" Oct 30 19:01:55.363514 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:01:55.363491 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a\": container with ID starting with 31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a not found: ID does not exist" containerID="31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a" Oct 30 19:01:55.363590 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.363525 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a"} err="failed to get container status \"31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a\": rpc error: code = NotFound desc = could not find container \"31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a\": container with ID starting with 31262619a4696fa308133f9f51b636410be4fc1bf7e7e9340de7ec2254488f9a not found: ID does not exist" Oct 30 19:01:55.427262 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.427232 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5"] Oct 30 19:01:55.461568 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:55.461533 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-qgxv5"] Oct 30 19:01:56.400739 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:56.400701 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:57.302150 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:57.302117 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" path="/var/lib/kubelet/pods/bed21b2d-f889-45c4-8d56-e5da10476136/volumes" Oct 30 19:01:58.376644 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:01:58.376609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:00.509708 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:00.509666 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:01.258575 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:01.258546 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:02.792731 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:02.792694 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:04.895529 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:04.895495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:06.226137 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:02:06.226097 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:02:07.128268 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:07.128232 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:09.495085 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:09.495047 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:11.516859 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:11.516807 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:13.654929 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:13.654891 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:14.087599 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:14.087512 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:16.154638 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:16.154609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:18.743762 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:18.743724 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:19.198271 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:19.198231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:19.537595 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:19.537507 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:21.020097 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:21.020028 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:22.147300 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:22.147260 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:23.770625 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:23.770593 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:24.517656 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:24.517617 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:26.117581 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:26.117543 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:28.042624 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:28.042591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:29.542541 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.542507 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:29.569042 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.569010 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j"] Oct 30 19:02:29.569235 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.569202 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" podUID="3e3b4189-6b83-428d-aa5c-63336edae122" containerName="maas-api" containerID="cri-o://b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031" gracePeriod=30 Oct 30 19:02:29.733983 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.733951 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:29.860381 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.860347 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 19:02:29.947195 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.947160 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-maas-api-tls\") pod \"3e3b4189-6b83-428d-aa5c-63336edae122\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " Oct 30 19:02:29.947405 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.947220 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-metrics-certs\") pod \"3e3b4189-6b83-428d-aa5c-63336edae122\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " Oct 30 19:02:29.947405 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.947374 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3b4189-6b83-428d-aa5c-63336edae122-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b4189-6b83-428d-aa5c-63336edae122-openshift-service-ca\") pod \"3e3b4189-6b83-428d-aa5c-63336edae122\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " Oct 30 19:02:29.947548 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.947495 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3b4189-6b83-428d-aa5c-63336edae122-kube-api-access-sbl2s\" (UniqueName: \"kubernetes.io/projected/3e3b4189-6b83-428d-aa5c-63336edae122-kube-api-access-sbl2s\") pod \"3e3b4189-6b83-428d-aa5c-63336edae122\" (UID: \"3e3b4189-6b83-428d-aa5c-63336edae122\") " Oct 30 19:02:29.947766 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.947741 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3b4189-6b83-428d-aa5c-63336edae122-openshift-service-ca" pod "3e3b4189-6b83-428d-aa5c-63336edae122" (UID: "3e3b4189-6b83-428d-aa5c-63336edae122"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:29.948218 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.948195 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e3b4189-6b83-428d-aa5c-63336edae122-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:29.949582 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.949555 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-maas-api-tls" pod "3e3b4189-6b83-428d-aa5c-63336edae122" (UID: "3e3b4189-6b83-428d-aa5c-63336edae122"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:29.949886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.949858 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-metrics-certs" pod "3e3b4189-6b83-428d-aa5c-63336edae122" (UID: "3e3b4189-6b83-428d-aa5c-63336edae122"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:29.949979 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:29.949917 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3b4189-6b83-428d-aa5c-63336edae122-kube-api-access-sbl2s" pod "3e3b4189-6b83-428d-aa5c-63336edae122" (UID: "3e3b4189-6b83-428d-aa5c-63336edae122"). InnerVolumeSpecName "kube-api-access-sbl2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:30.050240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.050206 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:30.050240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.050237 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e3b4189-6b83-428d-aa5c-63336edae122-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:30.050240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.050246 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbl2s\" (UniqueName: \"kubernetes.io/projected/3e3b4189-6b83-428d-aa5c-63336edae122-kube-api-access-sbl2s\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:30.603917 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.603881 4582 generic.go:356] "Generic (PLEG): container finished" podID="3e3b4189-6b83-428d-aa5c-63336edae122" containerID="b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031" exitCode=0 Oct 30 19:02:30.604384 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.603922 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" Oct 30 19:02:30.604384 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.603968 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" event={"ID":"3e3b4189-6b83-428d-aa5c-63336edae122","Type":"ContainerDied","Data":"b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031"} Oct 30 19:02:30.604384 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.604024 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j" event={"ID":"3e3b4189-6b83-428d-aa5c-63336edae122","Type":"ContainerDied","Data":"eb242cc18f96927c9c34f36888c2cc1383210e601953db4cd3a6b4837a262598"} Oct 30 19:02:30.604384 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.604048 4582 scope.go:122] "RemoveContainer" containerID="b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031" Oct 30 19:02:30.616258 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.616242 4582 scope.go:122] "RemoveContainer" containerID="b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031" Oct 30 19:02:30.616511 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:02:30.616496 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031\": container with ID starting with b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031 not found: ID does not exist" containerID="b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031" Oct 30 19:02:30.616555 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.616519 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031"} err="failed to get container status \"b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031\": rpc error: code = NotFound desc = could not find container \"b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031\": container with ID starting with b35a37f28e50153a819154ab4a4312bab1d00caea4c72528a421756cf86a7031 not found: ID does not exist" Oct 30 19:02:30.638878 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.638851 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:30.674676 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.674644 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j"] Oct 30 19:02:30.688492 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.688467 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-551d0b-7bc9db588f-vld2j"] Oct 30 19:02:30.842901 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:30.842868 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:31.290178 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:31.290138 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:31.302595 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:31.302562 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3b4189-6b83-428d-aa5c-63336edae122" path="/var/lib/kubelet/pods/3e3b4189-6b83-428d-aa5c-63336edae122/volumes" Oct 30 19:02:32.876773 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:32.876738 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:34.849429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:34.849385 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:36.225679 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:02:36.225640 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:02:36.686570 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:36.686533 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:37.868926 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:37.868891 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:37.968521 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:37.968471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:38.212741 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:38.212651 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6"] Oct 30 19:02:38.212968 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:38.212933 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" containerID="cri-o://6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd" gracePeriod=30 Oct 30 19:02:38.238785 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:38.238755 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:38.872062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:38.872027 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:40.275043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:40.274983 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:40.384401 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:40.384362 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:40.626407 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:40.626367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:40.732786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:40.732748 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:41.325838 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:41.325785 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:43.398874 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.398633 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:43.569067 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.568701 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 19:02:43.634298 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634266 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-data\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.634516 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634412 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-socket\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.634516 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634446 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-podinfo\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.634516 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634507 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-credential-socket\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.634709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634558 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-data" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.634709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634580 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-envoy\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.634709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634618 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25eeb43-e27f-48bc-844a-4411c76fed6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25eeb43-e27f-48bc-844a-4411c76fed6a-istiod-ca-cert\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.634709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634643 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-socket" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.634709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634688 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-kube-api-access-j7xcs\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-kube-api-access-j7xcs\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.635012 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634725 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-token\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-token\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.635012 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634780 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-certs\") pod \"f25eeb43-e27f-48bc-844a-4411c76fed6a\" (UID: \"f25eeb43-e27f-48bc-844a-4411c76fed6a\") " Oct 30 19:02:43.635012 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.634837 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-credential-socket" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.635172 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.635062 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25eeb43-e27f-48bc-844a-4411c76fed6a-istiod-ca-cert" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:43.635215 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.635185 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-certs" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.635483 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.635463 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.635594 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.635487 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.635594 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.635504 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.635594 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.635518 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f25eeb43-e27f-48bc-844a-4411c76fed6a-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.635594 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.635531 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.636856 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.636800 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-podinfo" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:02:43.637149 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.637133 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-envoy" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.637390 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.637369 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-token" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:43.637480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.637401 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-kube-api-access-j7xcs" pod "f25eeb43-e27f-48bc-844a-4411c76fed6a" (UID: "f25eeb43-e27f-48bc-844a-4411c76fed6a"). InnerVolumeSpecName "kube-api-access-j7xcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:43.728720 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.728682 4582 generic.go:356] "Generic (PLEG): container finished" podID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerID="6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd" exitCode=0 Oct 30 19:02:43.728929 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.728871 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" event={"ID":"f25eeb43-e27f-48bc-844a-4411c76fed6a","Type":"ContainerDied","Data":"6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd"} Oct 30 19:02:43.728929 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.728912 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" event={"ID":"f25eeb43-e27f-48bc-844a-4411c76fed6a","Type":"ContainerDied","Data":"a6486666955653ca6756f0f51d4d277ace65e967fe5b6ceb57f6d74f0dcc6813"} Oct 30 19:02:43.729062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.728933 4582 scope.go:122] "RemoveContainer" containerID="6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd" Oct 30 19:02:43.729062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.729052 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6" Oct 30 19:02:43.736670 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.736642 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.736777 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.736673 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7xcs\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-kube-api-access-j7xcs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.736777 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.736688 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.736777 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.736702 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f25eeb43-e27f-48bc-844a-4411c76fed6a-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.742227 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.742205 4582 scope.go:122] "RemoveContainer" containerID="6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd" Oct 30 19:02:43.742568 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:02:43.742540 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd\": container with ID starting with 6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd not found: ID does not exist" containerID="6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd" Oct 30 19:02:43.742647 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.742580 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd"} err="failed to get container status \"6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd\": rpc error: code = NotFound desc = could not find container \"6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd\": container with ID starting with 6339590aa4dc920f43ebe146b61544bdce2f2b55d5518bcc4262fa9cfb6a11dd not found: ID does not exist" Oct 30 19:02:43.789934 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.789813 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6"] Oct 30 19:02:43.793441 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:43.793413 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-4rzg6"] Oct 30 19:02:45.301690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:45.301647 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" path="/var/lib/kubelet/pods/f25eeb43-e27f-48bc-844a-4411c76fed6a/volumes" Oct 30 19:02:45.457588 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:45.457538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:47.112721 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:47.112687 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:48.893607 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:48.893576 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:50.635860 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:50.635811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:52.383071 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:52.383036 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:54.179971 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:54.179931 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:55.955897 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:55.955861 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:57.500404 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:57.500362 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:58.033939 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:58.033904 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:58.213949 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:58.213915 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:58.549886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:58.549850 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:58.858747 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:02:58.858649 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:00.411340 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:00.411251 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:02.184399 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:02.184363 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:03.890148 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:03.890116 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:04.711442 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:04.711407 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:04.950411 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:04.950381 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:05.130543 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:05.130496 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:05.251694 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:05.251662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:05.894076 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:05.894035 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:06.124023 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:06.123960 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:06.226863 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:03:06.226774 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:03:06.249251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:06.249219 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:07.823411 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:07.823362 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:09.636406 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:09.636367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:11.359015 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:11.358981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:13.120476 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:13.120437 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:14.790744 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:14.790699 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:16.469344 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:16.469310 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:17.550258 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:17.550224 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:18.147546 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:18.147507 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:19.177301 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:19.177265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:19.954541 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:19.954508 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:21.811037 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:21.811001 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:23.507575 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:23.507542 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:23.608084 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:23.608047 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:25.381502 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:25.381472 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:26.990566 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:26.990529 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:27.091111 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:27.091081 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:28.092335 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:28.092305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:28.879599 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:28.879556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:30.577300 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:30.577263 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:32.269187 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:32.269144 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:34.052881 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:34.052836 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:34.154596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:34.154562 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:35.777407 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:35.777372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:36.227526 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:03:36.227484 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:03:37.572768 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:37.572732 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:39.288693 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:39.288659 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:41.110739 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:41.110700 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:42.872477 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:42.872401 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:43.836474 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:43.836430 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:43.947088 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:43.947057 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:44.678495 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:44.678457 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:46.448273 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:46.448231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:48.185019 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:48.184989 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:49.957518 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:49.957482 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:51.645025 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:51.644991 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:53.468747 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:53.468715 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:55.232201 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:55.232166 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:57.071002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:57.070967 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:58.774428 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:03:58.774393 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:00.224591 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:00.224559 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:00.579886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:00.579783 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:02.278378 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:02.278343 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.968884 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:03.968816 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:05.736076 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:05.736044 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:06.227021 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:04:06.226986 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:04:06.691961 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:06.691919 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.647009 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:07.646976 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.359408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:09.359367 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:11.080116 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:11.080075 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:12.871571 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:12.871538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:14.562995 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:14.562958 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:16.295734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:16.295700 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:18.050312 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:18.050278 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:19.296414 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:19.296374 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:19.766947 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:19.766914 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:21.434079 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:21.434033 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:21.538912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:21.538855 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:23.355459 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:23.355420 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:25.142992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:25.142961 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:27.049605 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:27.049570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:28.870624 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:28.870589 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:30.649212 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:30.649174 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.387360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:32.387330 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:34.084183 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:34.084145 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:35.855992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:35.855948 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:36.227574 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:04:36.227537 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:04:37.587907 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:37.587867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:39.465310 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:39.465279 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:41.257003 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:41.256971 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:42.938983 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:42.938942 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:43.043430 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:43.043389 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:45.091832 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:45.091791 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:46.791614 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:46.791582 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:47.734313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:47.734254 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:47.837191 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:47.837159 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:47.941612 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:47.941548 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:48.983912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:48.983800 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:49.122647 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:49.122611 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:49.236664 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:49.236586 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:50.866783 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:50.866749 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:52.661194 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:52.661163 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:53.900574 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:53.900533 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:54.106989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:54.106957 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:54.514098 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:54.514059 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:55.267548 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:55.267512 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf"] Oct 30 19:04:55.289309 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:55.289277 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:56.471440 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:56.471411 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:58.224667 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:58.224633 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:58.424885 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:58.424849 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:58.883808 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:58.883770 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:59.088528 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:04:59.088493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:00.252937 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:00.252905 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:01.340730 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:01.340695 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:02.081583 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:02.081538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:03.816593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:03.816558 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:05.529549 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:05.529509 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:06.226632 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:05:06.226596 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:05:07.265784 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:07.265750 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:07.379428 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:07.379389 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:09.075668 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:09.075625 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:10.290205 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.290118 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" podUID="b78c648d-750c-4480-8cec-ab2aa12cc480" containerName="main" containerID="cri-o://4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8" gracePeriod=45 Oct 30 19:05:10.581314 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.581289 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:05:10.652055 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652020 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-home\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-home\") pod \"b78c648d-750c-4480-8cec-ab2aa12cc480\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " Oct 30 19:05:10.652251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652070 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78c648d-750c-4480-8cec-ab2aa12cc480-kube-api-access-qgvxx\" (UniqueName: \"kubernetes.io/projected/b78c648d-750c-4480-8cec-ab2aa12cc480-kube-api-access-qgvxx\") pod \"b78c648d-750c-4480-8cec-ab2aa12cc480\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " Oct 30 19:05:10.652251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652155 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-model-cache\") pod \"b78c648d-750c-4480-8cec-ab2aa12cc480\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " Oct 30 19:05:10.652251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652243 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-tmp-dir\") pod \"b78c648d-750c-4480-8cec-ab2aa12cc480\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " Oct 30 19:05:10.652453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652272 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-dshm\") pod \"b78c648d-750c-4480-8cec-ab2aa12cc480\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " Oct 30 19:05:10.652453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652302 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-home" pod "b78c648d-750c-4480-8cec-ab2aa12cc480" (UID: "b78c648d-750c-4480-8cec-ab2aa12cc480"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.652574 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652493 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-model-cache" pod "b78c648d-750c-4480-8cec-ab2aa12cc480" (UID: "b78c648d-750c-4480-8cec-ab2aa12cc480"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.652574 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652510 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-tmp-dir" pod "b78c648d-750c-4480-8cec-ab2aa12cc480" (UID: "b78c648d-750c-4480-8cec-ab2aa12cc480"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.652911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652890 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:10.653045 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652913 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:10.653045 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.652928 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:10.654644 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.654621 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78c648d-750c-4480-8cec-ab2aa12cc480-kube-api-access-qgvxx" pod "b78c648d-750c-4480-8cec-ab2aa12cc480" (UID: "b78c648d-750c-4480-8cec-ab2aa12cc480"). InnerVolumeSpecName "kube-api-access-qgvxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:10.654766 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.654734 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-dshm" pod "b78c648d-750c-4480-8cec-ab2aa12cc480" (UID: "b78c648d-750c-4480-8cec-ab2aa12cc480"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.749792 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.749762 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:10.753524 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.753501 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b78c648d-750c-4480-8cec-ab2aa12cc480-tls-certs\" (UniqueName: \"kubernetes.io/secret/b78c648d-750c-4480-8cec-ab2aa12cc480-tls-certs\") pod \"b78c648d-750c-4480-8cec-ab2aa12cc480\" (UID: \"b78c648d-750c-4480-8cec-ab2aa12cc480\") " Oct 30 19:05:10.753886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.753869 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgvxx\" (UniqueName: \"kubernetes.io/projected/b78c648d-750c-4480-8cec-ab2aa12cc480-kube-api-access-qgvxx\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:10.753973 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.753888 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b78c648d-750c-4480-8cec-ab2aa12cc480-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:10.755620 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.755598 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78c648d-750c-4480-8cec-ab2aa12cc480-tls-certs" pod "b78c648d-750c-4480-8cec-ab2aa12cc480" (UID: "b78c648d-750c-4480-8cec-ab2aa12cc480"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:10.800091 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.800009 4582 generic.go:356] "Generic (PLEG): container finished" podID="b78c648d-750c-4480-8cec-ab2aa12cc480" containerID="4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8" exitCode=0 Oct 30 19:05:10.800091 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.800054 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" Oct 30 19:05:10.800272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.800090 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" event={"ID":"b78c648d-750c-4480-8cec-ab2aa12cc480","Type":"ContainerDied","Data":"4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8"} Oct 30 19:05:10.800272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.800141 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf" event={"ID":"b78c648d-750c-4480-8cec-ab2aa12cc480","Type":"ContainerDied","Data":"9cc344343ef4108c36a7f7515f157d22350020a337a2f4800c888b9143c5cf57"} Oct 30 19:05:10.800272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.800159 4582 scope.go:122] "RemoveContainer" containerID="4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8" Oct 30 19:05:10.811796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.811776 4582 scope.go:122] "RemoveContainer" containerID="4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8" Oct 30 19:05:10.812123 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:05:10.812102 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8\": container with ID starting with 4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8 not found: ID does not exist" containerID="4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8" Oct 30 19:05:10.812194 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.812133 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8"} err="failed to get container status \"4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8\": rpc error: code = NotFound desc = could not find container \"4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8\": container with ID starting with 4fe82962f1a9ac3e03b8114c757e2fd78ba91aa56b8a8918ae00226e95cd97e8 not found: ID does not exist" Oct 30 19:05:10.853109 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.853074 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf"] Oct 30 19:05:10.855568 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.855542 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b78c648d-750c-4480-8cec-ab2aa12cc480-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:10.857430 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:10.857408 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-869b55/test-model-869b55-kserve-76b6d97df-zg8tf"] Oct 30 19:05:11.303752 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:11.303711 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78c648d-750c-4480-8cec-ab2aa12cc480" path="/var/lib/kubelet/pods/b78c648d-750c-4480-8cec-ab2aa12cc480/volumes" Oct 30 19:05:12.479611 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:12.479574 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:12.855889 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:12.855785 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:13.047762 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:13.047722 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:13.186765 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:13.186720 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:14.541608 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:14.541576 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:14.673418 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:14.673376 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:15.830654 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:15.830613 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:16.801950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:16.801912 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:18.286941 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:18.286909 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:19.436322 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:19.436288 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:20.179236 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:20.179197 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:21.917077 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:21.917042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:22.018590 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:22.018556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:23.796246 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:23.796212 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:25.629882 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:25.629846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:27.428165 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:27.428126 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:27.828167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:27.828082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:27.934891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:27.934842 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:29.315841 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:29.315783 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.423606 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:30.423570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.607798 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:30.607758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.749094 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:30.749013 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:31.172213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:31.172170 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:32.951024 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:32.950986 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:34.655754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:34.655718 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:36.131314 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.131280 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8"] Oct 30 19:05:36.132126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.132079 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" podUID="9b63d395-5cd1-4b4a-82d9-3d33b04fc502" containerName="maas-api" containerID="cri-o://68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e" gracePeriod=30 Oct 30 19:05:36.185655 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.185621 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:36.226508 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:05:36.226474 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:05:36.419847 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.419808 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 19:05:36.498906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.498871 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-metrics-certs\") pod \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " Oct 30 19:05:36.499106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.498946 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-maas-api-tls\") pod \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " Oct 30 19:05:36.499106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.498988 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-kube-api-access-5ptc9\" (UniqueName: \"kubernetes.io/projected/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-kube-api-access-5ptc9\") pod \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " Oct 30 19:05:36.499106 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.499092 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-openshift-service-ca\") pod \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\" (UID: \"9b63d395-5cd1-4b4a-82d9-3d33b04fc502\") " Oct 30 19:05:36.499740 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.499708 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-openshift-service-ca" pod "9b63d395-5cd1-4b4a-82d9-3d33b04fc502" (UID: "9b63d395-5cd1-4b4a-82d9-3d33b04fc502"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:36.501423 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.501394 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-maas-api-tls" pod "9b63d395-5cd1-4b4a-82d9-3d33b04fc502" (UID: "9b63d395-5cd1-4b4a-82d9-3d33b04fc502"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:36.501550 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.501469 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-metrics-certs" pod "9b63d395-5cd1-4b4a-82d9-3d33b04fc502" (UID: "9b63d395-5cd1-4b4a-82d9-3d33b04fc502"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:36.502434 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.502406 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-kube-api-access-5ptc9" pod "9b63d395-5cd1-4b4a-82d9-3d33b04fc502" (UID: "9b63d395-5cd1-4b4a-82d9-3d33b04fc502"). InnerVolumeSpecName "kube-api-access-5ptc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:36.600883 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.600851 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:36.600883 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.600880 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:36.601071 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.600893 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptc9\" (UniqueName: \"kubernetes.io/projected/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-kube-api-access-5ptc9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:36.601071 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.600902 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b63d395-5cd1-4b4a-82d9-3d33b04fc502-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:36.992915 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:36.992882 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:37.093400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.093327 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:37.107072 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.107039 4582 generic.go:356] "Generic (PLEG): container finished" podID="9b63d395-5cd1-4b4a-82d9-3d33b04fc502" containerID="68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e" exitCode=0 Oct 30 19:05:37.107222 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.107088 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" Oct 30 19:05:37.107222 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.107125 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" event={"ID":"9b63d395-5cd1-4b4a-82d9-3d33b04fc502","Type":"ContainerDied","Data":"68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e"} Oct 30 19:05:37.107222 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.107177 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8" event={"ID":"9b63d395-5cd1-4b4a-82d9-3d33b04fc502","Type":"ContainerDied","Data":"e263e018ad5f589f28050b9deca2a5eda4d759f751f8e548aaad22141d1eedc7"} Oct 30 19:05:37.107222 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.107200 4582 scope.go:122] "RemoveContainer" containerID="68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e" Oct 30 19:05:37.118848 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.118830 4582 scope.go:122] "RemoveContainer" containerID="68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e" Oct 30 19:05:37.119115 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:05:37.119096 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e\": container with ID starting with 68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e not found: ID does not exist" containerID="68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e" Oct 30 19:05:37.119166 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.119122 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e"} err="failed to get container status \"68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e\": rpc error: code = NotFound desc = could not find container \"68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e\": container with ID starting with 68a6405d0743924356de2507428d4eee98f2ffca4594cba0d447df4983576c7e not found: ID does not exist" Oct 30 19:05:37.159118 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.159079 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8"] Oct 30 19:05:37.170535 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.170513 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-869b55-8f8fd84bd-65gr8"] Oct 30 19:05:37.193604 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.193580 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:37.283892 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.283811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:37.303259 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.303230 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b63d395-5cd1-4b4a-82d9-3d33b04fc502" path="/var/lib/kubelet/pods/9b63d395-5cd1-4b4a-82d9-3d33b04fc502/volumes" Oct 30 19:05:37.369519 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.369485 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:37.775617 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:37.775557 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:38.259435 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.259396 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l"] Oct 30 19:05:38.260575 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260548 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b78c648d-750c-4480-8cec-ab2aa12cc480" containerName="main" Oct 30 19:05:38.260575 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260575 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78c648d-750c-4480-8cec-ab2aa12cc480" containerName="main" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260596 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260605 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260646 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3e3b4189-6b83-428d-aa5c-63336edae122" containerName="maas-api" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260653 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3b4189-6b83-428d-aa5c-63336edae122" containerName="maas-api" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260679 4582 cpu_manager.go:414] "containerMap: removing container" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260688 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260736 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9b63d395-5cd1-4b4a-82d9-3d33b04fc502" containerName="maas-api" Oct 30 19:05:38.260771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260744 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b63d395-5cd1-4b4a-82d9-3d33b04fc502" containerName="maas-api" Oct 30 19:05:38.261299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260922 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b63d395-5cd1-4b4a-82d9-3d33b04fc502" containerName="maas-api" Oct 30 19:05:38.261299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260941 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78c648d-750c-4480-8cec-ab2aa12cc480" containerName="main" Oct 30 19:05:38.261299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260952 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25eeb43-e27f-48bc-844a-4411c76fed6a" containerName="istio-proxy" Oct 30 19:05:38.261299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260966 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3b4189-6b83-428d-aa5c-63336edae122" containerName="maas-api" Oct 30 19:05:38.261299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.260977 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed21b2d-f889-45c4-8d56-e5da10476136" containerName="istio-proxy" Oct 30 19:05:38.266935 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.266912 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.280584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.280561 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l"] Oct 30 19:05:38.288692 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.288670 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-236fc4\"/\"openshift-service-ca.crt\"" Oct 30 19:05:38.288879 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.288861 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-236fc4\"/\"sub-test-model-236fc4-kserve-self-signed-certs\"" Oct 30 19:05:38.292986 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.292963 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-236fc4\"/\"kube-root-ca.crt\"" Oct 30 19:05:38.293103 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.292992 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-236fc4\"/\"default-dockercfg-9q4d4\"" Oct 30 19:05:38.308121 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.308094 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:38.322274 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.322252 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-model-cache\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.322394 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.322282 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-tmp-dir\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.322478 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.322457 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601ad845-dc26-4eb2-be78-4ef20d119b09-tls-certs\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.322595 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.322574 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-dshm\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.322717 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.322610 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkxl\" (UniqueName: \"kubernetes.io/projected/601ad845-dc26-4eb2-be78-4ef20d119b09-kube-api-access-gvkxl\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.322776 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.322750 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-home\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.423960 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.423919 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkxl\" (UniqueName: \"kubernetes.io/projected/601ad845-dc26-4eb2-be78-4ef20d119b09-kube-api-access-gvkxl\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.423976 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-home\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.424055 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-model-cache\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.424071 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-tmp-dir\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424261 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.424167 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601ad845-dc26-4eb2-be78-4ef20d119b09-tls-certs\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424298 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.424260 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-dshm\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424475 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.424448 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-home\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424513 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.424474 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-model-cache\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.424571 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.424549 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-tmp-dir\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.426492 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.426461 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-dshm\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.426921 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.426898 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601ad845-dc26-4eb2-be78-4ef20d119b09-tls-certs\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.442801 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.442772 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkxl\" (UniqueName: \"kubernetes.io/projected/601ad845-dc26-4eb2-be78-4ef20d119b09-kube-api-access-gvkxl\") pod \"sub-test-model-236fc4-kserve-58488d9669-jj98l\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.581971 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.581886 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:38.887148 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:38.887123 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l"] Oct 30 19:05:38.888011 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:05:38.887983 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod601ad845_dc26_4eb2_be78_4ef20d119b09.slice/crio-7ef35200b8480963e1bfc62b4c13d2e26fba4263001a4b177d394b831ad6b907 WatchSource:0}: Error finding container 7ef35200b8480963e1bfc62b4c13d2e26fba4263001a4b177d394b831ad6b907: Status 404 returned error can't find the container with id 7ef35200b8480963e1bfc62b4c13d2e26fba4263001a4b177d394b831ad6b907 Oct 30 19:05:39.123735 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:39.123692 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" event={"ID":"601ad845-dc26-4eb2-be78-4ef20d119b09","Type":"ContainerStarted","Data":"703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7"} Oct 30 19:05:39.123942 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:39.123745 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" event={"ID":"601ad845-dc26-4eb2-be78-4ef20d119b09","Type":"ContainerStarted","Data":"7ef35200b8480963e1bfc62b4c13d2e26fba4263001a4b177d394b831ad6b907"} Oct 30 19:05:39.168139 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:39.168044 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" podStartSLOduration=1.168031015 podStartE2EDuration="1.168031015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:39.166297362 +0000 UTC m=+4035.990691675" watchObservedRunningTime="2026-10-30 19:05:39.168031015 +0000 UTC m=+4035.992425306" Oct 30 19:05:39.177509 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:39.177455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:39.582919 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:39.582843 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:40.982193 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:40.982152 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:42.684461 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:42.684426 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:44.127922 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:44.127886 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:44.241792 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:44.241761 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:44.454923 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:44.454306 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:44.552784 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:44.552749 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:46.135032 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.134997 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv"] Oct 30 19:05:46.135466 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.135196 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" podUID="b2b904d1-097d-4a1a-a5d6-6367b4a426dd" containerName="maas-api" containerID="cri-o://dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95" gracePeriod=30 Oct 30 19:05:46.261592 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.261566 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:46.416199 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.416171 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 19:05:46.527324 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.527286 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-maas-api-tls\") pod \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " Oct 30 19:05:46.527484 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.527387 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-metrics-certs\") pod \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " Oct 30 19:05:46.527484 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.527459 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-openshift-service-ca\") pod \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " Oct 30 19:05:46.527580 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.527501 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-kube-api-access-f46tk\" (UniqueName: \"kubernetes.io/projected/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-kube-api-access-f46tk\") pod \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\" (UID: \"b2b904d1-097d-4a1a-a5d6-6367b4a426dd\") " Oct 30 19:05:46.527883 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.527810 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-openshift-service-ca" pod "b2b904d1-097d-4a1a-a5d6-6367b4a426dd" (UID: "b2b904d1-097d-4a1a-a5d6-6367b4a426dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:46.528230 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.528209 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:46.529579 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.529547 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-metrics-certs" pod "b2b904d1-097d-4a1a-a5d6-6367b4a426dd" (UID: "b2b904d1-097d-4a1a-a5d6-6367b4a426dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:46.529690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.529627 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-maas-api-tls" pod "b2b904d1-097d-4a1a-a5d6-6367b4a426dd" (UID: "b2b904d1-097d-4a1a-a5d6-6367b4a426dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:46.529690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.529648 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-kube-api-access-f46tk" pod "b2b904d1-097d-4a1a-a5d6-6367b4a426dd" (UID: "b2b904d1-097d-4a1a-a5d6-6367b4a426dd"). InnerVolumeSpecName "kube-api-access-f46tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:46.629796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.629725 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46tk\" (UniqueName: \"kubernetes.io/projected/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-kube-api-access-f46tk\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:46.629796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.629770 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:46.629796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:46.629785 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2b904d1-097d-4a1a-a5d6-6367b4a426dd-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:05:47.152625 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.152584 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:47.188107 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.188076 4582 generic.go:356] "Generic (PLEG): container finished" podID="b2b904d1-097d-4a1a-a5d6-6367b4a426dd" containerID="dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95" exitCode=0 Oct 30 19:05:47.188262 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.188125 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" Oct 30 19:05:47.188262 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.188164 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" event={"ID":"b2b904d1-097d-4a1a-a5d6-6367b4a426dd","Type":"ContainerDied","Data":"dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95"} Oct 30 19:05:47.188262 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.188224 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv" event={"ID":"b2b904d1-097d-4a1a-a5d6-6367b4a426dd","Type":"ContainerDied","Data":"b202ab7d6959da70df5ccd7e6a6562ec66f08d090729971b1963479d3bf16ae0"} Oct 30 19:05:47.188262 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.188248 4582 scope.go:122] "RemoveContainer" containerID="dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95" Oct 30 19:05:47.199888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.199866 4582 scope.go:122] "RemoveContainer" containerID="dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95" Oct 30 19:05:47.200194 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:05:47.200168 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95\": container with ID starting with dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95 not found: ID does not exist" containerID="dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95" Oct 30 19:05:47.200267 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.200202 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95"} err="failed to get container status \"dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95\": rpc error: code = NotFound desc = could not find container \"dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95\": container with ID starting with dd54757fcf6fb7f09362a1e7ce85edbe6f6bbcb597aad553524f824810b42e95 not found: ID does not exist" Oct 30 19:05:47.242188 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.242155 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv"] Oct 30 19:05:47.253633 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.253600 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2820d9-74984cc9b4-d5zbv"] Oct 30 19:05:47.272704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.272643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:47.303228 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.303200 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b904d1-097d-4a1a-a5d6-6367b4a426dd" path="/var/lib/kubelet/pods/b2b904d1-097d-4a1a-a5d6-6367b4a426dd/volumes" Oct 30 19:05:47.374145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.374101 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:47.472976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.472899 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:47.813975 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:47.813900 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:48.582968 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:48.582935 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:48.595709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:48.595684 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:49.150946 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:49.150906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:49.220021 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:49.219993 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:05:50.829781 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:50.829742 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:52.989131 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:52.989090 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:54.325920 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:54.325885 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:54.437789 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:54.437761 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:54.702294 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:54.682044 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:56.107610 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:56.107570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:56.208533 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:56.208494 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:56.309897 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:56.309861 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:58.167553 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:58.167523 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:59.898389 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:05:59.898356 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:01.759175 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:01.759141 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:03.455795 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:03.455761 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:05.175047 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:05.175008 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:06.226580 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:06:06.226543 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:06:06.456604 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:06.456568 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:06.572623 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:06.572540 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:06.598610 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:06.598570 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6"] Oct 30 19:06:06.598834 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:06.598799 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" podUID="c596df49-8d89-4082-a5d2-d8f720f8c584" containerName="maas-api" containerID="cri-o://012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc" gracePeriod=30 Oct 30 19:06:06.668754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:06.668710 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:06.904008 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:06.903981 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 19:06:07.009680 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.009644 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c596df49-8d89-4082-a5d2-d8f720f8c584-kube-api-access-qnspt\" (UniqueName: \"kubernetes.io/projected/c596df49-8d89-4082-a5d2-d8f720f8c584-kube-api-access-qnspt\") pod \"c596df49-8d89-4082-a5d2-d8f720f8c584\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " Oct 30 19:06:07.009891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.009849 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-metrics-certs\") pod \"c596df49-8d89-4082-a5d2-d8f720f8c584\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " Oct 30 19:06:07.009970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.009905 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-maas-api-tls\") pod \"c596df49-8d89-4082-a5d2-d8f720f8c584\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " Oct 30 19:06:07.010045 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.009971 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c596df49-8d89-4082-a5d2-d8f720f8c584-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c596df49-8d89-4082-a5d2-d8f720f8c584-openshift-service-ca\") pod \"c596df49-8d89-4082-a5d2-d8f720f8c584\" (UID: \"c596df49-8d89-4082-a5d2-d8f720f8c584\") " Oct 30 19:06:07.010352 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.010323 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c596df49-8d89-4082-a5d2-d8f720f8c584-openshift-service-ca" pod "c596df49-8d89-4082-a5d2-d8f720f8c584" (UID: "c596df49-8d89-4082-a5d2-d8f720f8c584"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:07.010528 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.010509 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c596df49-8d89-4082-a5d2-d8f720f8c584-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:07.011981 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.011955 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c596df49-8d89-4082-a5d2-d8f720f8c584-kube-api-access-qnspt" pod "c596df49-8d89-4082-a5d2-d8f720f8c584" (UID: "c596df49-8d89-4082-a5d2-d8f720f8c584"). InnerVolumeSpecName "kube-api-access-qnspt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:07.012213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.011999 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-maas-api-tls" pod "c596df49-8d89-4082-a5d2-d8f720f8c584" (UID: "c596df49-8d89-4082-a5d2-d8f720f8c584"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:07.012213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.012034 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-metrics-certs" pod "c596df49-8d89-4082-a5d2-d8f720f8c584" (UID: "c596df49-8d89-4082-a5d2-d8f720f8c584"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:07.111956 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.111906 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:07.111956 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.111948 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c596df49-8d89-4082-a5d2-d8f720f8c584-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:07.111956 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.111962 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnspt\" (UniqueName: \"kubernetes.io/projected/c596df49-8d89-4082-a5d2-d8f720f8c584-kube-api-access-qnspt\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:07.333710 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.333675 4582 generic.go:356] "Generic (PLEG): container finished" podID="c596df49-8d89-4082-a5d2-d8f720f8c584" containerID="012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc" exitCode=0 Oct 30 19:06:07.334153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.333733 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" Oct 30 19:06:07.334153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.333759 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" event={"ID":"c596df49-8d89-4082-a5d2-d8f720f8c584","Type":"ContainerDied","Data":"012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc"} Oct 30 19:06:07.334153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.333810 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6" event={"ID":"c596df49-8d89-4082-a5d2-d8f720f8c584","Type":"ContainerDied","Data":"fe67e4b0f83f6c2e1c837124d390d9b84c061dfcc42c74f4870dd1e46fcd8718"} Oct 30 19:06:07.334153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.333848 4582 scope.go:122] "RemoveContainer" containerID="012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc" Oct 30 19:06:07.346435 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.346409 4582 scope.go:122] "RemoveContainer" containerID="012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc" Oct 30 19:06:07.346885 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:06:07.346856 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc\": container with ID starting with 012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc not found: ID does not exist" containerID="012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc" Oct 30 19:06:07.346996 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.346888 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc"} err="failed to get container status \"012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc\": rpc error: code = NotFound desc = could not find container \"012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc\": container with ID starting with 012a89f4aada7aa57f7a5db6e7d65170e4fd863d191d540198bef90ca74ef4bc not found: ID does not exist" Oct 30 19:06:07.384530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.384502 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6"] Oct 30 19:06:07.396200 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.396178 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-daf3ba-6b6485c79c-848b6"] Oct 30 19:06:07.463541 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.463506 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:07.531246 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.531219 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm"] Oct 30 19:06:07.531937 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.531917 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c596df49-8d89-4082-a5d2-d8f720f8c584" containerName="maas-api" Oct 30 19:06:07.531999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.531938 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c596df49-8d89-4082-a5d2-d8f720f8c584" containerName="maas-api" Oct 30 19:06:07.531999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.531980 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b2b904d1-097d-4a1a-a5d6-6367b4a426dd" containerName="maas-api" Oct 30 19:06:07.531999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.531986 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b904d1-097d-4a1a-a5d6-6367b4a426dd" containerName="maas-api" Oct 30 19:06:07.532134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.532091 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b904d1-097d-4a1a-a5d6-6367b4a426dd" containerName="maas-api" Oct 30 19:06:07.532134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.532107 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c596df49-8d89-4082-a5d2-d8f720f8c584" containerName="maas-api" Oct 30 19:06:07.537205 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.537189 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.555168 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.555143 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-15e8827b-serving-cert\"" Oct 30 19:06:07.556610 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.556583 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm"] Oct 30 19:06:07.583385 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.583312 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:07.616206 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.616173 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f4c09d-0328-4b20-8e83-56b4a31a0c61-openshift-service-ca\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.616421 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.616403 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.616482 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.616428 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-metrics-certs\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.616482 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.616443 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvlcz\" (UniqueName: \"kubernetes.io/projected/63f4c09d-0328-4b20-8e83-56b4a31a0c61-kube-api-access-gvlcz\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.670766 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.670735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:07.717387 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.717351 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f4c09d-0328-4b20-8e83-56b4a31a0c61-openshift-service-ca\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.717555 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.717538 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.717649 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.717582 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-metrics-certs\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.717649 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.717610 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvlcz\" (UniqueName: \"kubernetes.io/projected/63f4c09d-0328-4b20-8e83-56b4a31a0c61-kube-api-access-gvlcz\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.717808 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:06:07.717694 4582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-serving-cert: secret "maas-api-e2e-mt-15e8827b-serving-cert" not found Oct 30 19:06:07.717808 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:06:07.717760 4582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls podName:63f4c09d-0328-4b20-8e83-56b4a31a0c61 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:08.21774166 +0000 UTC m=+4065.042135929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls") pod "maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" (UID: "63f4c09d-0328-4b20-8e83-56b4a31a0c61") : secret "maas-api-e2e-mt-15e8827b-serving-cert" not found Oct 30 19:06:07.718133 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.718116 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f4c09d-0328-4b20-8e83-56b4a31a0c61-openshift-service-ca\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.719991 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.719970 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-metrics-certs\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.737696 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.737672 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvlcz\" (UniqueName: \"kubernetes.io/projected/63f4c09d-0328-4b20-8e83-56b4a31a0c61-kube-api-access-gvlcz\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:07.806515 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:07.806480 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:08.110277 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:08.110231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:08.223932 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:08.223896 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:08.226097 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:08.226075 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls\") pod \"maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:08.451476 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:08.451439 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:08.756698 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:08.756645 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm"] Oct 30 19:06:08.759258 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:08.759232 4582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:06:09.301480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:09.301446 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c596df49-8d89-4082-a5d2-d8f720f8c584" path="/var/lib/kubelet/pods/c596df49-8d89-4082-a5d2-d8f720f8c584/volumes" Oct 30 19:06:09.350790 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:09.350753 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" event={"ID":"63f4c09d-0328-4b20-8e83-56b4a31a0c61","Type":"ContainerStarted","Data":"bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2"} Oct 30 19:06:09.351021 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:09.350800 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" event={"ID":"63f4c09d-0328-4b20-8e83-56b4a31a0c61","Type":"ContainerStarted","Data":"a7eb674d7da3b6bd12ead675e94ebae4c16aac2a5364850f986515cc8b531f4d"} Oct 30 19:06:09.351021 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:09.350855 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:09.393317 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:09.393263 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" podStartSLOduration=2.007475491 podStartE2EDuration="2.393248186s" totalImagesPullingTime="385.772695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:09.390553946 +0000 UTC m=+4066.214948227" watchObservedRunningTime="2026-10-30 19:06:09.393248186 +0000 UTC m=+4066.217642479" Oct 30 19:06:09.617365 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:09.617275 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:11.370767 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:11.361043 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:13.062181 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:13.062145 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:14.427783 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:14.427741 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:14.531596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:14.531565 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:14.803690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:14.803606 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:15.035496 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:15.035454 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:15.363741 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:15.363714 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:06:15.455310 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:15.455272 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:16.974948 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:16.974910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:18.752600 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:18.752513 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:19.516374 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:19.516314 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:20.583989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:20.583946 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:22.321370 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:22.321335 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:24.029729 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:24.029688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:25.676069 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:25.676034 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:27.384931 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:27.384889 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:29.129293 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:29.129247 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:30.940940 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:30.940910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:31.488011 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:31.487978 4582 scope.go:122] "RemoveContainer" containerID="363be81559d3cbefd8a5384127889d8dc7385d1e862ae855900f99934ad3d75a" Oct 30 19:06:32.977218 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:32.977187 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:34.773256 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:34.773221 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:36.227519 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:06:36.227486 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:06:36.760173 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:36.760133 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:38.689959 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:38.689906 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:40.550943 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:40.550896 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:41.489568 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:41.489535 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:42.361170 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:42.361127 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:42.462452 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:42.462414 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:44.059477 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:44.059444 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:45.819810 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:45.819777 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:47.569499 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:47.569461 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.141975 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:49.141926 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.421597 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:49.421502 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:51.156950 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:51.156911 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:52.878160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:52.878118 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:54.392054 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.392012 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6"] Oct 30 19:06:54.392656 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.392257 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" podUID="f53c015c-5a4f-4d06-95dd-5ff15c9e225f" containerName="maas-api" containerID="cri-o://d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e" gracePeriod=30 Oct 30 19:06:54.442436 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.442404 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:54.680847 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.680799 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 19:06:54.681690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.681663 4582 generic.go:356] "Generic (PLEG): container finished" podID="f53c015c-5a4f-4d06-95dd-5ff15c9e225f" containerID="d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e" exitCode=0 Oct 30 19:06:54.681851 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.681742 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" event={"ID":"f53c015c-5a4f-4d06-95dd-5ff15c9e225f","Type":"ContainerDied","Data":"d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e"} Oct 30 19:06:54.681939 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.681924 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" event={"ID":"f53c015c-5a4f-4d06-95dd-5ff15c9e225f","Type":"ContainerDied","Data":"2b4c94b8ffdf85c431f7f2d10c47a4054655b3a1b8179664ff10ffc3ca3b0e6e"} Oct 30 19:06:54.682036 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.681957 4582 scope.go:122] "RemoveContainer" containerID="d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e" Oct 30 19:06:54.692383 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.692362 4582 scope.go:122] "RemoveContainer" containerID="d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e" Oct 30 19:06:54.692659 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:06:54.692636 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e\": container with ID starting with d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e not found: ID does not exist" containerID="d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e" Oct 30 19:06:54.692712 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.692667 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e"} err="failed to get container status \"d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e\": rpc error: code = NotFound desc = could not find container \"d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e\": container with ID starting with d57219891eed23b9b8704eb42d4189dae5851a7985c26718bd73da623ab28b8e not found: ID does not exist" Oct 30 19:06:54.748594 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.748562 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-openshift-service-ca\") pod \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " Oct 30 19:06:54.748786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.748608 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-maas-api-tls\") pod \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " Oct 30 19:06:54.748786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.748637 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-metrics-certs\") pod \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " Oct 30 19:06:54.748786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.748694 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-kube-api-access-dww8d\" (UniqueName: \"kubernetes.io/projected/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-kube-api-access-dww8d\") pod \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\" (UID: \"f53c015c-5a4f-4d06-95dd-5ff15c9e225f\") " Oct 30 19:06:54.749042 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.749011 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-openshift-service-ca" pod "f53c015c-5a4f-4d06-95dd-5ff15c9e225f" (UID: "f53c015c-5a4f-4d06-95dd-5ff15c9e225f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:54.749847 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.749792 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:54.751174 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.751150 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-metrics-certs" pod "f53c015c-5a4f-4d06-95dd-5ff15c9e225f" (UID: "f53c015c-5a4f-4d06-95dd-5ff15c9e225f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:54.751267 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.751224 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-kube-api-access-dww8d" pod "f53c015c-5a4f-4d06-95dd-5ff15c9e225f" (UID: "f53c015c-5a4f-4d06-95dd-5ff15c9e225f"). InnerVolumeSpecName "kube-api-access-dww8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:54.752028 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.752005 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-maas-api-tls" pod "f53c015c-5a4f-4d06-95dd-5ff15c9e225f" (UID: "f53c015c-5a4f-4d06-95dd-5ff15c9e225f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:54.850841 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.850786 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:54.850841 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.850814 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:54.850841 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:54.850841 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dww8d\" (UniqueName: \"kubernetes.io/projected/f53c015c-5a4f-4d06-95dd-5ff15c9e225f-kube-api-access-dww8d\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:06:55.146213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:55.146178 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:55.271720 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:55.271687 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:55.360994 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:55.360964 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:55.689490 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:55.689462 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6" Oct 30 19:06:55.741691 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:55.741664 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6"] Oct 30 19:06:55.755047 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:55.755020 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f6b7c-797b678cfc-wmcj6"] Oct 30 19:06:55.800448 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:55.800422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:57.252713 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:57.252680 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:57.302522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:57.302495 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53c015c-5a4f-4d06-95dd-5ff15c9e225f" path="/var/lib/kubelet/pods/f53c015c-5a4f-4d06-95dd-5ff15c9e225f/volumes" Oct 30 19:06:59.040209 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:59.040172 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:59.173512 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:59.173478 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:59.273573 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:59.273535 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:59.818999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:06:59.818957 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:00.751566 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:00.751530 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:00.940752 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:00.940708 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:01.140809 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:01.140770 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:01.393988 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:01.393901 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:02.943580 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:02.943548 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:03.046829 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:03.046794 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:03.148709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:03.148677 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:03.305165 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:03.305089 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:04.272387 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:04.272355 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:04.377309 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:04.377267 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:04.595466 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:04.595369 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:05.165248 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:05.165204 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:06.227309 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:07:06.227273 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:07:06.873563 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:06.873517 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:08.119335 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:08.119280 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:08.676665 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:08.676626 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:10.420801 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:10.420755 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:10.517494 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:10.517440 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:12.713890 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:12.713849 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:15.735198 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:15.735145 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:17.721307 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:17.721275 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:19.634564 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:19.634531 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:19.737455 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:19.737421 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:21.302960 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:21.302928 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:21.403050 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:21.403015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:21.761975 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:21.761937 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:23.576630 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:23.576592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:25.357410 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:25.357372 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:27.111530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:27.111497 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:28.757042 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:28.757002 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:29.090319 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:29.090241 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:29.188949 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:29.188920 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:30.458441 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.458407 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9"] Oct 30 19:07:30.458900 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.458667 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" podUID="f0063e82-afb7-46df-9f43-324a55f24992" containerName="maas-api" containerID="cri-o://58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a" gracePeriod=30 Oct 30 19:07:30.669131 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.669092 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:30.735951 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.735922 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 19:07:30.826904 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.826867 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-metrics-certs\") pod \"f0063e82-afb7-46df-9f43-324a55f24992\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " Oct 30 19:07:30.827114 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.826932 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0063e82-afb7-46df-9f43-324a55f24992-kube-api-access-bvm22\" (UniqueName: \"kubernetes.io/projected/f0063e82-afb7-46df-9f43-324a55f24992-kube-api-access-bvm22\") pod \"f0063e82-afb7-46df-9f43-324a55f24992\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " Oct 30 19:07:30.827114 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.827001 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-maas-api-tls\") pod \"f0063e82-afb7-46df-9f43-324a55f24992\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " Oct 30 19:07:30.827261 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.827168 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0063e82-afb7-46df-9f43-324a55f24992-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0063e82-afb7-46df-9f43-324a55f24992-openshift-service-ca\") pod \"f0063e82-afb7-46df-9f43-324a55f24992\" (UID: \"f0063e82-afb7-46df-9f43-324a55f24992\") " Oct 30 19:07:30.828026 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.827995 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0063e82-afb7-46df-9f43-324a55f24992-openshift-service-ca" pod "f0063e82-afb7-46df-9f43-324a55f24992" (UID: "f0063e82-afb7-46df-9f43-324a55f24992"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:30.829618 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.829590 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0063e82-afb7-46df-9f43-324a55f24992-kube-api-access-bvm22" pod "f0063e82-afb7-46df-9f43-324a55f24992" (UID: "f0063e82-afb7-46df-9f43-324a55f24992"). InnerVolumeSpecName "kube-api-access-bvm22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:30.829778 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.829751 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-metrics-certs" pod "f0063e82-afb7-46df-9f43-324a55f24992" (UID: "f0063e82-afb7-46df-9f43-324a55f24992"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:30.830297 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.830271 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-maas-api-tls" pod "f0063e82-afb7-46df-9f43-324a55f24992" (UID: "f0063e82-afb7-46df-9f43-324a55f24992"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:30.928731 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.928701 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0063e82-afb7-46df-9f43-324a55f24992-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:30.928731 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.928730 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:30.928939 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.928739 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvm22\" (UniqueName: \"kubernetes.io/projected/f0063e82-afb7-46df-9f43-324a55f24992-kube-api-access-bvm22\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:30.928939 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.928750 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0063e82-afb7-46df-9f43-324a55f24992-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:30.989628 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.989534 4582 generic.go:356] "Generic (PLEG): container finished" podID="f0063e82-afb7-46df-9f43-324a55f24992" containerID="58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a" exitCode=0 Oct 30 19:07:30.989814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.989591 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" Oct 30 19:07:30.989814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.989643 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" event={"ID":"f0063e82-afb7-46df-9f43-324a55f24992","Type":"ContainerDied","Data":"58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a"} Oct 30 19:07:30.989814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.989718 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9" event={"ID":"f0063e82-afb7-46df-9f43-324a55f24992","Type":"ContainerDied","Data":"ac9fe2e995e0ad2ca3d1c443c96b40060f0d3c091e1409be758484bdc07af02d"} Oct 30 19:07:30.989814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:30.989754 4582 scope.go:122] "RemoveContainer" containerID="58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a" Oct 30 19:07:31.001795 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.001773 4582 scope.go:122] "RemoveContainer" containerID="58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a" Oct 30 19:07:31.002130 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:07:31.002108 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a\": container with ID starting with 58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a not found: ID does not exist" containerID="58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a" Oct 30 19:07:31.002184 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.002137 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a"} err="failed to get container status \"58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a\": rpc error: code = NotFound desc = could not find container \"58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a\": container with ID starting with 58ebb19a05efdf6369a6fe9e937d42ee8450337aef99a631b68c2248ba99541a not found: ID does not exist" Oct 30 19:07:31.040991 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.040962 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9"] Oct 30 19:07:31.053055 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.053031 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-861bb3-76d5cdcd59-sgdl9"] Oct 30 19:07:31.295569 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.295490 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:31.301760 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.301736 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0063e82-afb7-46df-9f43-324a55f24992" path="/var/lib/kubelet/pods/f0063e82-afb7-46df-9f43-324a55f24992/volumes" Oct 30 19:07:31.395214 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.395189 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:31.485868 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.485797 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:31.921423 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:31.921384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:33.337769 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:33.337720 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:34.978233 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:34.978196 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:36.229466 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:07:36.229427 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:07:36.779277 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:36.779238 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:38.113789 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:38.113755 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:38.316328 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:38.316290 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:38.524090 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:38.524060 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:38.607611 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:38.607572 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:40.485090 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:40.485053 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:42.283506 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:42.283452 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:44.059382 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:44.059345 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:45.771901 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:45.771863 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:47.627310 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:47.627265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:49.425647 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:49.425555 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:49.731813 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:49.731727 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:49.834628 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:49.834593 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:51.286492 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:51.286460 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:52.837756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:52.837713 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq"] Oct 30 19:07:52.838276 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:52.837934 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" podUID="fdc61542-a9b2-4137-a526-4aa71dffe281" containerName="maas-api" containerID="cri-o://4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02" gracePeriod=30 Oct 30 19:07:52.884399 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:52.884357 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:53.114952 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.114881 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 19:07:53.139259 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.139225 4582 generic.go:356] "Generic (PLEG): container finished" podID="fdc61542-a9b2-4137-a526-4aa71dffe281" containerID="4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02" exitCode=0 Oct 30 19:07:53.139437 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.139267 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" Oct 30 19:07:53.139437 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.139316 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" event={"ID":"fdc61542-a9b2-4137-a526-4aa71dffe281","Type":"ContainerDied","Data":"4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02"} Oct 30 19:07:53.139437 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.139373 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq" event={"ID":"fdc61542-a9b2-4137-a526-4aa71dffe281","Type":"ContainerDied","Data":"69e27a5702a960c0d48de75b306b6213130ae6670b170a6e7c1c61016af3f7ed"} Oct 30 19:07:53.139437 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.139399 4582 scope.go:122] "RemoveContainer" containerID="4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02" Oct 30 19:07:53.151124 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.151101 4582 scope.go:122] "RemoveContainer" containerID="4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02" Oct 30 19:07:53.151512 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:07:53.151491 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02\": container with ID starting with 4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02 not found: ID does not exist" containerID="4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02" Oct 30 19:07:53.151611 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.151521 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02"} err="failed to get container status \"4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02\": rpc error: code = NotFound desc = could not find container \"4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02\": container with ID starting with 4efd0534cfde059cbef984326cbbce0fdbaa2769f903bc4d44ff91767165bd02 not found: ID does not exist" Oct 30 19:07:53.215584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.215558 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc61542-a9b2-4137-a526-4aa71dffe281-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc61542-a9b2-4137-a526-4aa71dffe281-openshift-service-ca\") pod \"fdc61542-a9b2-4137-a526-4aa71dffe281\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " Oct 30 19:07:53.215746 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.215607 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-maas-api-tls\") pod \"fdc61542-a9b2-4137-a526-4aa71dffe281\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " Oct 30 19:07:53.215746 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.215695 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-metrics-certs\") pod \"fdc61542-a9b2-4137-a526-4aa71dffe281\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " Oct 30 19:07:53.215864 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.215770 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc61542-a9b2-4137-a526-4aa71dffe281-kube-api-access-54vb6\" (UniqueName: \"kubernetes.io/projected/fdc61542-a9b2-4137-a526-4aa71dffe281-kube-api-access-54vb6\") pod \"fdc61542-a9b2-4137-a526-4aa71dffe281\" (UID: \"fdc61542-a9b2-4137-a526-4aa71dffe281\") " Oct 30 19:07:53.215994 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.215971 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc61542-a9b2-4137-a526-4aa71dffe281-openshift-service-ca" pod "fdc61542-a9b2-4137-a526-4aa71dffe281" (UID: "fdc61542-a9b2-4137-a526-4aa71dffe281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:53.216345 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.216321 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc61542-a9b2-4137-a526-4aa71dffe281-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:53.217781 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.217752 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc61542-a9b2-4137-a526-4aa71dffe281-kube-api-access-54vb6" pod "fdc61542-a9b2-4137-a526-4aa71dffe281" (UID: "fdc61542-a9b2-4137-a526-4aa71dffe281"). InnerVolumeSpecName "kube-api-access-54vb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:53.217781 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.217766 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-metrics-certs" pod "fdc61542-a9b2-4137-a526-4aa71dffe281" (UID: "fdc61542-a9b2-4137-a526-4aa71dffe281"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:53.217942 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.217785 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-maas-api-tls" pod "fdc61542-a9b2-4137-a526-4aa71dffe281" (UID: "fdc61542-a9b2-4137-a526-4aa71dffe281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:53.320213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.317974 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:53.320213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.318008 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc61542-a9b2-4137-a526-4aa71dffe281-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:53.320213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.318021 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vb6\" (UniqueName: \"kubernetes.io/projected/fdc61542-a9b2-4137-a526-4aa71dffe281-kube-api-access-54vb6\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:07:53.479297 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.479265 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq"] Oct 30 19:07:53.490565 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.490538 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6018b4-665795b455-28jcq"] Oct 30 19:07:53.589699 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.589666 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:53.754700 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.754622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:53.862123 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:53.862091 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:54.281753 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:54.281723 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:55.302162 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:55.302128 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc61542-a9b2-4137-a526-4aa71dffe281" path="/var/lib/kubelet/pods/fdc61542-a9b2-4137-a526-4aa71dffe281/volumes" Oct 30 19:07:55.681188 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:55.681156 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:57.484248 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:57.484212 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:59.226209 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:07:59.226178 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:01.006075 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:01.006042 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:01.113066 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:01.113029 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:01.247245 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:01.247209 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:01.467427 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:01.467388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:02.902449 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:02.902399 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:04.555830 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:04.555781 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:05.596988 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:05.596951 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:05.698928 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:05.698893 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:05.794943 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:05.794903 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:06.227276 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:08:06.227239 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:08:06.233129 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:06.233103 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:07.676332 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:07.676291 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:09.311796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:09.311765 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:10.988598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:10.988565 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:12.513526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:12.513492 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d"] Oct 30 19:08:12.546751 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:12.546708 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh"] Oct 30 19:08:12.687321 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:12.687281 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:12.798797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:12.798722 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:12.910402 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:12.910365 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:13.776970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:13.776938 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:13.877795 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:13.877762 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:14.120290 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:14.120261 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:14.763074 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:14.763033 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:15.570005 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:15.569973 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:16.874470 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:16.874436 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:17.513954 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:17.513900 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" podUID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" containerName="payload-processing" containerID="cri-o://0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7" gracePeriod=25 Oct 30 19:08:17.547156 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:17.547109 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" podUID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" containerName="payload-pre-processing" containerID="cri-o://30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003" gracePeriod=25 Oct 30 19:08:18.855360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:18.855316 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:19.733340 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:19.733305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:20.468356 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:20.468320 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" podUID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:20.468842 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:20.468329 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" podUID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:20.675659 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:20.675618 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:22.492731 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:22.492700 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:24.240643 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:24.240604 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:25.321155 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:25.321118 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:25.426686 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:25.426652 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:26.308073 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:26.308028 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:28.133998 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:28.133962 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:29.797108 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:29.797067 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:30.468397 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:30.468359 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" podUID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:30.468576 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:30.468359 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" podUID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:31.268740 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.268702 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz"] Oct 30 19:08:31.269230 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.268970 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" podUID="af692e1e-ce02-464f-8ae5-637403b9a7b0" containerName="maas-api" containerID="cri-o://a0f70aab0cd6233705b33bab3410bf7a62f0ee084de9d60fe5f30a640c28a6c9" gracePeriod=30 Oct 30 19:08:31.323006 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.322967 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:31.397796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.397760 4582 generic.go:356] "Generic (PLEG): container finished" podID="af692e1e-ce02-464f-8ae5-637403b9a7b0" containerID="a0f70aab0cd6233705b33bab3410bf7a62f0ee084de9d60fe5f30a640c28a6c9" exitCode=0 Oct 30 19:08:31.397925 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.397854 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" event={"ID":"af692e1e-ce02-464f-8ae5-637403b9a7b0","Type":"ContainerDied","Data":"a0f70aab0cd6233705b33bab3410bf7a62f0ee084de9d60fe5f30a640c28a6c9"} Oct 30 19:08:31.550758 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.550731 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 19:08:31.570137 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.559815 4582 scope.go:122] "RemoveContainer" containerID="a0f70aab0cd6233705b33bab3410bf7a62f0ee084de9d60fe5f30a640c28a6c9" Oct 30 19:08:31.616858 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.616799 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-metrics-certs\") pod \"af692e1e-ce02-464f-8ae5-637403b9a7b0\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " Oct 30 19:08:31.617048 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.617033 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af692e1e-ce02-464f-8ae5-637403b9a7b0-kube-api-access-8xh7r\" (UniqueName: \"kubernetes.io/projected/af692e1e-ce02-464f-8ae5-637403b9a7b0-kube-api-access-8xh7r\") pod \"af692e1e-ce02-464f-8ae5-637403b9a7b0\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " Oct 30 19:08:31.617163 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.617149 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af692e1e-ce02-464f-8ae5-637403b9a7b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af692e1e-ce02-464f-8ae5-637403b9a7b0-openshift-service-ca\") pod \"af692e1e-ce02-464f-8ae5-637403b9a7b0\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " Oct 30 19:08:31.617213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.617179 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls\") pod \"af692e1e-ce02-464f-8ae5-637403b9a7b0\" (UID: \"af692e1e-ce02-464f-8ae5-637403b9a7b0\") " Oct 30 19:08:31.617560 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.617529 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af692e1e-ce02-464f-8ae5-637403b9a7b0-openshift-service-ca" pod "af692e1e-ce02-464f-8ae5-637403b9a7b0" (UID: "af692e1e-ce02-464f-8ae5-637403b9a7b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:31.617691 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.617674 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af692e1e-ce02-464f-8ae5-637403b9a7b0-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:31.619251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.619222 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-metrics-certs" pod "af692e1e-ce02-464f-8ae5-637403b9a7b0" (UID: "af692e1e-ce02-464f-8ae5-637403b9a7b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:31.619352 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.619304 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af692e1e-ce02-464f-8ae5-637403b9a7b0-kube-api-access-8xh7r" pod "af692e1e-ce02-464f-8ae5-637403b9a7b0" (UID: "af692e1e-ce02-464f-8ae5-637403b9a7b0"). InnerVolumeSpecName "kube-api-access-8xh7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:31.619678 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.619662 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls" pod "af692e1e-ce02-464f-8ae5-637403b9a7b0" (UID: "af692e1e-ce02-464f-8ae5-637403b9a7b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:31.718760 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.718727 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:31.718760 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.718756 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af692e1e-ce02-464f-8ae5-637403b9a7b0-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:31.718760 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.718766 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xh7r\" (UniqueName: \"kubernetes.io/projected/af692e1e-ce02-464f-8ae5-637403b9a7b0-kube-api-access-8xh7r\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:31.989191 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:31.989161 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:32.111968 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:32.111930 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:32.250163 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:32.250088 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:32.405501 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:32.405460 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" event={"ID":"af692e1e-ce02-464f-8ae5-637403b9a7b0","Type":"ContainerDied","Data":"b09dbee5ca8ae54f10c97cf7d3a6e81fc9b22e469d12a53b28e0ecb884c790b7"} Oct 30 19:08:32.406052 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:32.405517 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz" Oct 30 19:08:32.485141 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:32.485105 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz"] Oct 30 19:08:32.489357 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:32.489328 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-38e0db-f48cc8798-mbhvz"] Oct 30 19:08:32.822209 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:32.822177 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:33.301801 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:33.301766 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af692e1e-ce02-464f-8ae5-637403b9a7b0" path="/var/lib/kubelet/pods/af692e1e-ce02-464f-8ae5-637403b9a7b0/volumes" Oct 30 19:08:33.543065 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:33.543026 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:33.642866 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:33.642812 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:34.185031 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:34.184991 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:35.725226 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:35.725188 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct"] Oct 30 19:08:35.725755 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:35.725441 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" podUID="9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" containerName="maas-api" containerID="cri-o://d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3" gracePeriod=30 Oct 30 19:08:35.862049 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:35.862020 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.010223 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.010196 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 19:08:36.075107 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.075075 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-openshift-service-ca\") pod \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " Oct 30 19:08:36.075327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.075190 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-metrics-certs\") pod \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " Oct 30 19:08:36.075327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.075228 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-maas-api-tls\") pod \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " Oct 30 19:08:36.075327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.075295 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-kube-api-access-h9hcl\" (UniqueName: \"kubernetes.io/projected/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-kube-api-access-h9hcl\") pod \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\" (UID: \"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd\") " Oct 30 19:08:36.076785 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.076750 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-openshift-service-ca" pod "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" (UID: "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:36.078730 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.078704 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-kube-api-access-h9hcl" pod "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" (UID: "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd"). InnerVolumeSpecName "kube-api-access-h9hcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:36.082300 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.082261 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-maas-api-tls" pod "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" (UID: "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:36.083071 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.083038 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-metrics-certs" pod "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" (UID: "9fbf9d38-6d75-43d5-8d16-74b3cafb40dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:36.177453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.177411 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:36.177453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.177450 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:36.177453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.177460 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:36.177722 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.177469 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hcl\" (UniqueName: \"kubernetes.io/projected/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd-kube-api-access-h9hcl\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:36.227524 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:08:36.227488 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:08:36.433305 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.433267 4582 generic.go:356] "Generic (PLEG): container finished" podID="9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" containerID="d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3" exitCode=0 Oct 30 19:08:36.433504 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.433321 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" Oct 30 19:08:36.433504 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.433357 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" event={"ID":"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd","Type":"ContainerDied","Data":"d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3"} Oct 30 19:08:36.433504 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.433412 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct" event={"ID":"9fbf9d38-6d75-43d5-8d16-74b3cafb40dd","Type":"ContainerDied","Data":"5394c420fd7998a195f23fda0749d290ccca62f393364ab39894718f985f7e45"} Oct 30 19:08:36.433504 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.433438 4582 scope.go:122] "RemoveContainer" containerID="d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3" Oct 30 19:08:36.447280 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.447259 4582 scope.go:122] "RemoveContainer" containerID="d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3" Oct 30 19:08:36.447640 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:08:36.447608 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3\": container with ID starting with d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3 not found: ID does not exist" containerID="d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3" Oct 30 19:08:36.447751 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.447649 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3"} err="failed to get container status \"d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3\": rpc error: code = NotFound desc = could not find container \"d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3\": container with ID starting with d15439640583842e1c3ff627afc2aebaf5ad17eff19384c44fd12bb3f1887ac3 not found: ID does not exist" Oct 30 19:08:36.484458 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.484424 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct"] Oct 30 19:08:36.496082 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.496051 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e95c51-69b4887dc4-2glct"] Oct 30 19:08:36.621656 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.621616 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.724079 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.723998 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.821642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:36.821608 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:37.302221 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:37.302188 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" path="/var/lib/kubelet/pods/9fbf9d38-6d75-43d5-8d16-74b3cafb40dd/volumes" Oct 30 19:08:37.679275 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:08:37.679239 4582 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0845c6d_ab7d_4077_a5ba_a6d151b3272a.slice/crio-30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0845c6d_ab7d_4077_a5ba_a6d151b3272a.slice/crio-conmon-30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:37.844035 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:37.844009 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 19:08:37.860555 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:37.860495 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 19:08:38.002132 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002092 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1890c4b-70b6-4783-b73f-fc6d58634d1a-kube-api-access-b8m5t\" (UniqueName: \"kubernetes.io/projected/b1890c4b-70b6-4783-b73f-fc6d58634d1a-kube-api-access-b8m5t\") pod \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " Oct 30 19:08:38.002334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002152 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1890c4b-70b6-4783-b73f-fc6d58634d1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1890c4b-70b6-4783-b73f-fc6d58634d1a-plugins-config-volume\") pod \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\" (UID: \"b1890c4b-70b6-4783-b73f-fc6d58634d1a\") " Oct 30 19:08:38.002334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002195 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-plugins-config-volume\") pod \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " Oct 30 19:08:38.002458 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002413 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-kube-api-access-vqxp5\" (UniqueName: \"kubernetes.io/projected/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-kube-api-access-vqxp5\") pod \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\" (UID: \"c0845c6d-ab7d-4077-a5ba-a6d151b3272a\") " Oct 30 19:08:38.002563 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002535 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1890c4b-70b6-4783-b73f-fc6d58634d1a-plugins-config-volume" pod "b1890c4b-70b6-4783-b73f-fc6d58634d1a" (UID: "b1890c4b-70b6-4783-b73f-fc6d58634d1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:38.002627 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002590 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-plugins-config-volume" pod "c0845c6d-ab7d-4077-a5ba-a6d151b3272a" (UID: "c0845c6d-ab7d-4077-a5ba-a6d151b3272a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:38.002931 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002913 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1890c4b-70b6-4783-b73f-fc6d58634d1a-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:38.002931 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.002930 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:38.004384 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.004358 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1890c4b-70b6-4783-b73f-fc6d58634d1a-kube-api-access-b8m5t" pod "b1890c4b-70b6-4783-b73f-fc6d58634d1a" (UID: "b1890c4b-70b6-4783-b73f-fc6d58634d1a"). InnerVolumeSpecName "kube-api-access-b8m5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:38.004494 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.004426 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-kube-api-access-vqxp5" pod "c0845c6d-ab7d-4077-a5ba-a6d151b3272a" (UID: "c0845c6d-ab7d-4077-a5ba-a6d151b3272a"). InnerVolumeSpecName "kube-api-access-vqxp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:38.104722 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.104692 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqxp5\" (UniqueName: \"kubernetes.io/projected/c0845c6d-ab7d-4077-a5ba-a6d151b3272a-kube-api-access-vqxp5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:38.104722 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.104718 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8m5t\" (UniqueName: \"kubernetes.io/projected/b1890c4b-70b6-4783-b73f-fc6d58634d1a-kube-api-access-b8m5t\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:08:38.448587 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.448549 4582 generic.go:356] "Generic (PLEG): container finished" podID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" containerID="0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7" exitCode=0 Oct 30 19:08:38.448775 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.448596 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" Oct 30 19:08:38.448775 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.448636 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" event={"ID":"b1890c4b-70b6-4783-b73f-fc6d58634d1a","Type":"ContainerDied","Data":"0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7"} Oct 30 19:08:38.448775 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.448692 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d" event={"ID":"b1890c4b-70b6-4783-b73f-fc6d58634d1a","Type":"ContainerDied","Data":"22b7d4e7cf2b253586517a5aa76b67c78422231a5fcd52e13dfffb8ffc276729"} Oct 30 19:08:38.448775 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.448720 4582 scope.go:122] "RemoveContainer" containerID="0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7" Oct 30 19:08:38.450115 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.450090 4582 generic.go:356] "Generic (PLEG): container finished" podID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" containerID="30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003" exitCode=0 Oct 30 19:08:38.450252 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.450141 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" Oct 30 19:08:38.450252 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.450192 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" event={"ID":"c0845c6d-ab7d-4077-a5ba-a6d151b3272a","Type":"ContainerDied","Data":"30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003"} Oct 30 19:08:38.450357 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.450253 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh" event={"ID":"c0845c6d-ab7d-4077-a5ba-a6d151b3272a","Type":"ContainerDied","Data":"b8e71b9e9b1bddca3af7841948134c5b8009a7ad6792fd2d47cb86c1ced4c723"} Oct 30 19:08:38.460257 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.460238 4582 scope.go:122] "RemoveContainer" containerID="0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7" Oct 30 19:08:38.460519 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:08:38.460502 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7\": container with ID starting with 0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7 not found: ID does not exist" containerID="0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7" Oct 30 19:08:38.460579 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.460527 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7"} err="failed to get container status \"0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7\": rpc error: code = NotFound desc = could not find container \"0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7\": container with ID starting with 0b4a0090258321994626e0b92b4799c8996b6c27b563b740f9677fb0211cb6f7 not found: ID does not exist" Oct 30 19:08:38.460579 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.460544 4582 scope.go:122] "RemoveContainer" containerID="30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003" Oct 30 19:08:38.469568 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.469551 4582 scope.go:122] "RemoveContainer" containerID="30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003" Oct 30 19:08:38.469856 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:08:38.469837 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003\": container with ID starting with 30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003 not found: ID does not exist" containerID="30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003" Oct 30 19:08:38.469923 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.469863 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003"} err="failed to get container status \"30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003\": rpc error: code = NotFound desc = could not find container \"30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003\": container with ID starting with 30dd5890ff73b5239d39334db3626feda057e2b40e7aa8c91784e61bd7059003 not found: ID does not exist" Oct 30 19:08:38.498502 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.498467 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh"] Oct 30 19:08:38.510360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.510333 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e95c51-7958f9fdcf-798lh"] Oct 30 19:08:38.536193 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.536165 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d"] Oct 30 19:08:38.539128 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.539102 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e95c51-6b68cbff46-ms29d"] Oct 30 19:08:38.680118 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:38.680082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:39.300977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:39.300945 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" path="/var/lib/kubelet/pods/b1890c4b-70b6-4783-b73f-fc6d58634d1a/volumes" Oct 30 19:08:39.301399 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:39.301385 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" path="/var/lib/kubelet/pods/c0845c6d-ab7d-4077-a5ba-a6d151b3272a/volumes" Oct 30 19:08:39.583765 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:39.583691 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:39.686543 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:39.686508 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:39.933567 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:39.933534 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:40.823434 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:40.823398 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:42.696732 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:42.696695 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:42.780066 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:42.780026 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:44.473598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:44.473568 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:46.299869 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:46.299811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:47.228575 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:47.228539 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:48.069098 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:48.069028 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:49.834071 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:49.834030 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:50.729936 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.729899 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv"] Oct 30 19:08:50.731046 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731023 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" containerName="maas-api" Oct 30 19:08:50.731160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731047 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" containerName="maas-api" Oct 30 19:08:50.731160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731077 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f53c015c-5a4f-4d06-95dd-5ff15c9e225f" containerName="maas-api" Oct 30 19:08:50.731160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731086 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53c015c-5a4f-4d06-95dd-5ff15c9e225f" containerName="maas-api" Oct 30 19:08:50.731160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731119 4582 cpu_manager.go:414] "containerMap: removing container" podUID="fdc61542-a9b2-4137-a526-4aa71dffe281" containerName="maas-api" Oct 30 19:08:50.731160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731127 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc61542-a9b2-4137-a526-4aa71dffe281" containerName="maas-api" Oct 30 19:08:50.731160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731141 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" containerName="payload-processing" Oct 30 19:08:50.731160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731149 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" containerName="payload-processing" Oct 30 19:08:50.731398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731168 4582 cpu_manager.go:414] "containerMap: removing container" podUID="f0063e82-afb7-46df-9f43-324a55f24992" containerName="maas-api" Oct 30 19:08:50.731398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731176 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0063e82-afb7-46df-9f43-324a55f24992" containerName="maas-api" Oct 30 19:08:50.731398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731192 4582 cpu_manager.go:414] "containerMap: removing container" podUID="af692e1e-ce02-464f-8ae5-637403b9a7b0" containerName="maas-api" Oct 30 19:08:50.731398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731199 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af692e1e-ce02-464f-8ae5-637403b9a7b0" containerName="maas-api" Oct 30 19:08:50.731398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731253 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" containerName="payload-pre-processing" Oct 30 19:08:50.731398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731261 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" containerName="payload-pre-processing" Oct 30 19:08:50.731593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731420 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0063e82-afb7-46df-9f43-324a55f24992" containerName="maas-api" Oct 30 19:08:50.731593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731431 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="af692e1e-ce02-464f-8ae5-637403b9a7b0" containerName="maas-api" Oct 30 19:08:50.731593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731453 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0845c6d-ab7d-4077-a5ba-a6d151b3272a" containerName="payload-pre-processing" Oct 30 19:08:50.731593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731465 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbf9d38-6d75-43d5-8d16-74b3cafb40dd" containerName="maas-api" Oct 30 19:08:50.731593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731480 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53c015c-5a4f-4d06-95dd-5ff15c9e225f" containerName="maas-api" Oct 30 19:08:50.731593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731493 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc61542-a9b2-4137-a526-4aa71dffe281" containerName="maas-api" Oct 30 19:08:50.731593 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.731502 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1890c4b-70b6-4783-b73f-fc6d58634d1a" containerName="payload-processing" Oct 30 19:08:50.736026 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.736004 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.754511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.754485 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv"] Oct 30 19:08:50.761029 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.761010 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"openshift-service-ca.crt\"" Oct 30 19:08:50.761170 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.761108 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"kube-root-ca.crt\"" Oct 30 19:08:50.762809 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.762781 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"default-dockercfg-chvpr\"" Oct 30 19:08:50.762948 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.762931 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-24b124\"/\"rate-test-model-24b124-kserve-self-signed-certs\"" Oct 30 19:08:50.774864 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.774834 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tmp-dir\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.774972 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.774888 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvjq\" (UniqueName: \"kubernetes.io/projected/19653dd4-9014-4a62-a060-97a9dd1e6ba9-kube-api-access-pcvjq\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.774972 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.774965 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tls-certs\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.775103 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.775074 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-home\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.775103 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.775091 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-dshm\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.775229 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.775197 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-model-cache\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.783357 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.783337 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:50.875944 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.875907 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-home\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.875944 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.875949 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-dshm\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.876510 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.876020 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-model-cache\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.876510 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.876126 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tmp-dir\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.876510 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.876204 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvjq\" (UniqueName: \"kubernetes.io/projected/19653dd4-9014-4a62-a060-97a9dd1e6ba9-kube-api-access-pcvjq\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.876510 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.876247 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tls-certs\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.876510 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.876412 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-home\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.876510 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.876429 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-model-cache\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.876873 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.876654 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tmp-dir\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.878429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.878409 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-dshm\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.879168 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.879148 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tls-certs\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:50.895866 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:50.895843 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvjq\" (UniqueName: \"kubernetes.io/projected/19653dd4-9014-4a62-a060-97a9dd1e6ba9-kube-api-access-pcvjq\") pod \"rate-test-model-24b124-kserve-5989f7f745-89gvv\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:51.063575 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.063484 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:51.370838 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.370798 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv"] Oct 30 19:08:51.372500 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:08:51.372470 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19653dd4_9014_4a62_a060_97a9dd1e6ba9.slice/crio-9e89c01e6fb95f26445a27ce4c4bee3b3f13e6800b05aa5fbdd0b55b4f57185c WatchSource:0}: Error finding container 9e89c01e6fb95f26445a27ce4c4bee3b3f13e6800b05aa5fbdd0b55b4f57185c: Status 404 returned error can't find the container with id 9e89c01e6fb95f26445a27ce4c4bee3b3f13e6800b05aa5fbdd0b55b4f57185c Oct 30 19:08:51.539638 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.539587 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" event={"ID":"19653dd4-9014-4a62-a060-97a9dd1e6ba9","Type":"ContainerStarted","Data":"88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc"} Oct 30 19:08:51.539911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.539645 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" event={"ID":"19653dd4-9014-4a62-a060-97a9dd1e6ba9","Type":"ContainerStarted","Data":"9e89c01e6fb95f26445a27ce4c4bee3b3f13e6800b05aa5fbdd0b55b4f57185c"} Oct 30 19:08:51.580902 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.580853 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" podStartSLOduration=1.580840249 podStartE2EDuration="1.580840249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:51.578265838 +0000 UTC m=+4228.402660141" watchObservedRunningTime="2026-10-30 19:08:51.580840249 +0000 UTC m=+4228.405234540" Oct 30 19:08:51.688280 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.688195 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:51.825070 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.825035 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42"] Oct 30 19:08:51.986562 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:51.986476 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:52.064442 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:52.064412 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:08:52.086504 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:52.086473 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:53.463631 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:53.463599 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:53.563755 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:53.563704 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:53.897127 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:53.897090 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:55.459711 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:55.459678 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:56.825947 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:56.825912 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" podUID="5b02e285-5f9b-4702-b7a1-666dec453d4b" containerName="payload-processing" containerID="cri-o://a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51" gracePeriod=25 Oct 30 19:08:57.237314 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:57.237264 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:58.958094 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:08:58.958060 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:00.682491 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:00.682456 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:01.063868 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:01.063767 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:09:01.076383 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:01.076355 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:09:01.614207 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:01.614170 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:09:02.536426 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:02.536392 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:04.182436 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:04.182402 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:05.079498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:05.079459 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:05.182005 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:05.181961 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:05.889171 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:05.889131 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" podUID="5b02e285-5f9b-4702-b7a1-666dec453d4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:06.084511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:06.084482 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:06.228713 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:09:06.228628 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:09:07.950375 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:07.950334 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:09.744648 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:09.744607 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:10.097532 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:10.097443 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:11.548794 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:11.548748 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:12.981927 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:12.981893 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm"] Oct 30 19:09:12.982388 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:12.982130 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" podUID="63f4c09d-0328-4b20-8e83-56b4a31a0c61" containerName="maas-api" containerID="cri-o://bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2" gracePeriod=30 Oct 30 19:09:13.037159 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.037119 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:13.273531 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.273502 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:09:13.464189 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.464150 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls\") pod \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " Oct 30 19:09:13.464382 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.464285 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-metrics-certs\") pod \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " Oct 30 19:09:13.464382 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.464347 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f4c09d-0328-4b20-8e83-56b4a31a0c61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f4c09d-0328-4b20-8e83-56b4a31a0c61-openshift-service-ca\") pod \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " Oct 30 19:09:13.464546 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.464521 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f4c09d-0328-4b20-8e83-56b4a31a0c61-kube-api-access-gvlcz\" (UniqueName: \"kubernetes.io/projected/63f4c09d-0328-4b20-8e83-56b4a31a0c61-kube-api-access-gvlcz\") pod \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\" (UID: \"63f4c09d-0328-4b20-8e83-56b4a31a0c61\") " Oct 30 19:09:13.465011 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.464983 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f4c09d-0328-4b20-8e83-56b4a31a0c61-openshift-service-ca" pod "63f4c09d-0328-4b20-8e83-56b4a31a0c61" (UID: "63f4c09d-0328-4b20-8e83-56b4a31a0c61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:13.465264 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.465244 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f4c09d-0328-4b20-8e83-56b4a31a0c61-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:13.466966 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.466936 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls" pod "63f4c09d-0328-4b20-8e83-56b4a31a0c61" (UID: "63f4c09d-0328-4b20-8e83-56b4a31a0c61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:13.467094 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.467019 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-metrics-certs" pod "63f4c09d-0328-4b20-8e83-56b4a31a0c61" (UID: "63f4c09d-0328-4b20-8e83-56b4a31a0c61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:13.467163 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.467125 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f4c09d-0328-4b20-8e83-56b4a31a0c61-kube-api-access-gvlcz" pod "63f4c09d-0328-4b20-8e83-56b4a31a0c61" (UID: "63f4c09d-0328-4b20-8e83-56b4a31a0c61"). InnerVolumeSpecName "kube-api-access-gvlcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:13.567137 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.567051 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:13.567137 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.567081 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f4c09d-0328-4b20-8e83-56b4a31a0c61-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:13.567137 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.567095 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvlcz\" (UniqueName: \"kubernetes.io/projected/63f4c09d-0328-4b20-8e83-56b4a31a0c61-kube-api-access-gvlcz\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:13.693579 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.693543 4582 generic.go:356] "Generic (PLEG): container finished" podID="63f4c09d-0328-4b20-8e83-56b4a31a0c61" containerID="bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2" exitCode=0 Oct 30 19:09:13.693756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.693587 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" Oct 30 19:09:13.693756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.693629 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" event={"ID":"63f4c09d-0328-4b20-8e83-56b4a31a0c61","Type":"ContainerDied","Data":"bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2"} Oct 30 19:09:13.693756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.693687 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm" event={"ID":"63f4c09d-0328-4b20-8e83-56b4a31a0c61","Type":"ContainerDied","Data":"a7eb674d7da3b6bd12ead675e94ebae4c16aac2a5364850f986515cc8b531f4d"} Oct 30 19:09:13.693756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.693715 4582 scope.go:122] "RemoveContainer" containerID="bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2" Oct 30 19:09:13.704647 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.704629 4582 scope.go:122] "RemoveContainer" containerID="bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2" Oct 30 19:09:13.704985 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:09:13.704964 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2\": container with ID starting with bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2 not found: ID does not exist" containerID="bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2" Oct 30 19:09:13.705076 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.704990 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2"} err="failed to get container status \"bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2\": rpc error: code = NotFound desc = could not find container \"bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2\": container with ID starting with bae8b3d4b8d72dceedb8488aef22f99cf05dd0429e8283f915bd114c1dc316c2 not found: ID does not exist" Oct 30 19:09:13.711567 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.711527 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:13.744410 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.744385 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm"] Oct 30 19:09:13.756262 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.756239 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e8827b-7788d9d8c9-rhmnm"] Oct 30 19:09:13.819444 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.819369 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:13.901806 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:13.901774 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:14.326665 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:14.326631 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:15.301635 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:15.301600 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f4c09d-0328-4b20-8e83-56b4a31a0c61" path="/var/lib/kubelet/pods/63f4c09d-0328-4b20-8e83-56b4a31a0c61/volumes" Oct 30 19:09:15.787027 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:15.786993 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:15.889284 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:15.889242 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" podUID="5b02e285-5f9b-4702-b7a1-666dec453d4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:17.010756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.010733 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 19:09:17.105995 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.105918 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b02e285-5f9b-4702-b7a1-666dec453d4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b02e285-5f9b-4702-b7a1-666dec453d4b-plugins-config-volume\") pod \"5b02e285-5f9b-4702-b7a1-666dec453d4b\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " Oct 30 19:09:17.105995 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.105965 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b02e285-5f9b-4702-b7a1-666dec453d4b-kube-api-access-8qxxd\" (UniqueName: \"kubernetes.io/projected/5b02e285-5f9b-4702-b7a1-666dec453d4b-kube-api-access-8qxxd\") pod \"5b02e285-5f9b-4702-b7a1-666dec453d4b\" (UID: \"5b02e285-5f9b-4702-b7a1-666dec453d4b\") " Oct 30 19:09:17.106287 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.106264 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b02e285-5f9b-4702-b7a1-666dec453d4b-plugins-config-volume" pod "5b02e285-5f9b-4702-b7a1-666dec453d4b" (UID: "5b02e285-5f9b-4702-b7a1-666dec453d4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:17.106419 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.106406 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b02e285-5f9b-4702-b7a1-666dec453d4b-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:17.108204 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.108171 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b02e285-5f9b-4702-b7a1-666dec453d4b-kube-api-access-8qxxd" pod "5b02e285-5f9b-4702-b7a1-666dec453d4b" (UID: "5b02e285-5f9b-4702-b7a1-666dec453d4b"). InnerVolumeSpecName "kube-api-access-8qxxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:17.207584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.207540 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxxd\" (UniqueName: \"kubernetes.io/projected/5b02e285-5f9b-4702-b7a1-666dec453d4b-kube-api-access-8qxxd\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:17.505135 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.505101 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:17.723831 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.723795 4582 generic.go:356] "Generic (PLEG): container finished" podID="5b02e285-5f9b-4702-b7a1-666dec453d4b" containerID="a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51" exitCode=0 Oct 30 19:09:17.724043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.723862 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" event={"ID":"5b02e285-5f9b-4702-b7a1-666dec453d4b","Type":"ContainerDied","Data":"a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51"} Oct 30 19:09:17.724043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.723890 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" Oct 30 19:09:17.724043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.723932 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42" event={"ID":"5b02e285-5f9b-4702-b7a1-666dec453d4b","Type":"ContainerDied","Data":"361d654af294c09ca6e28b79a34bb090e42a6ba26e69d7bfe751bc033eeddaf3"} Oct 30 19:09:17.724043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.723970 4582 scope.go:122] "RemoveContainer" containerID="a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51" Oct 30 19:09:17.733793 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.733776 4582 scope.go:122] "RemoveContainer" containerID="a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51" Oct 30 19:09:17.734096 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:09:17.734072 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51\": container with ID starting with a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51 not found: ID does not exist" containerID="a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51" Oct 30 19:09:17.734205 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.734101 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51"} err="failed to get container status \"a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51\": rpc error: code = NotFound desc = could not find container \"a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51\": container with ID starting with a6e8ba063ddfb76b6f62ecd835824fbe851c21b2623fc7a1029d487889e3bc51 not found: ID does not exist" Oct 30 19:09:17.773294 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.773236 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42"] Oct 30 19:09:17.794798 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:17.785961 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ab244c-58d8bddb46-wzn42"] Oct 30 19:09:19.174257 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:19.174177 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:19.301804 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:19.301772 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b02e285-5f9b-4702-b7a1-666dec453d4b" path="/var/lib/kubelet/pods/5b02e285-5f9b-4702-b7a1-666dec453d4b/volumes" Oct 30 19:09:19.855135 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:19.855103 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:20.447271 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:20.447238 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:20.547642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:20.547605 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:20.766170 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:20.766089 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:20.877801 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:20.877764 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:22.042168 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.042136 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm"] Oct 30 19:09:22.042914 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.042893 4582 cpu_manager.go:414] "containerMap: removing container" podUID="63f4c09d-0328-4b20-8e83-56b4a31a0c61" containerName="maas-api" Oct 30 19:09:22.042991 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.042916 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f4c09d-0328-4b20-8e83-56b4a31a0c61" containerName="maas-api" Oct 30 19:09:22.043033 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.042991 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5b02e285-5f9b-4702-b7a1-666dec453d4b" containerName="payload-processing" Oct 30 19:09:22.043033 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.043001 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b02e285-5f9b-4702-b7a1-666dec453d4b" containerName="payload-processing" Oct 30 19:09:22.043206 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.043192 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b02e285-5f9b-4702-b7a1-666dec453d4b" containerName="payload-processing" Oct 30 19:09:22.043247 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.043218 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f4c09d-0328-4b20-8e83-56b4a31a0c61" containerName="maas-api" Oct 30 19:09:22.048904 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.048880 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.061924 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.061899 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm"] Oct 30 19:09:22.070307 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.070283 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"kube-root-ca.crt\"" Oct 30 19:09:22.070439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.070302 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"openshift-service-ca.crt\"" Oct 30 19:09:22.074584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.074556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:22.076051 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.076032 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"rate-test-model-675070-kserve-self-signed-certs\"" Oct 30 19:09:22.076111 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.076078 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-675070\"/\"default-dockercfg-xwr88\"" Oct 30 19:09:22.171451 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.171417 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-dshm\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.171627 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.171465 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-model-cache\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.171627 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.171559 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-tmp-dir\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.171705 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.171657 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbp5\" (UniqueName: \"kubernetes.io/projected/7835c137-0861-48e2-bb3f-ea72ef3060da-kube-api-access-flbp5\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.171705 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.171691 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7835c137-0861-48e2-bb3f-ea72ef3060da-tls-certs\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.171785 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.171707 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-home\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273143 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273095 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-model-cache\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273361 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273177 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-tmp-dir\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273361 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273287 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbp5\" (UniqueName: \"kubernetes.io/projected/7835c137-0861-48e2-bb3f-ea72ef3060da-kube-api-access-flbp5\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273361 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273334 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7835c137-0861-48e2-bb3f-ea72ef3060da-tls-certs\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273361 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273359 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-home\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273609 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273514 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-dshm\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273678 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273634 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-model-cache\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273678 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273658 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-tmp-dir\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.273799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.273767 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-home\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.275846 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.275774 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-dshm\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.276235 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.276215 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7835c137-0861-48e2-bb3f-ea72ef3060da-tls-certs\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.292685 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.292623 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbp5\" (UniqueName: \"kubernetes.io/projected/7835c137-0861-48e2-bb3f-ea72ef3060da-kube-api-access-flbp5\") pod \"rate-test-model-675070-kserve-7559848f57-zfgnm\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.363042 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.363010 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:22.477433 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.477335 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc"] Oct 30 19:09:22.477695 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.477582 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" podUID="cad19fb9-c8d7-48f0-9d05-992cafb88bbc" containerName="maas-api" containerID="cri-o://61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69" gracePeriod=30 Oct 30 19:09:22.533755 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.533720 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:22.669884 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.669850 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm"] Oct 30 19:09:22.670202 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:09:22.670177 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7835c137_0861_48e2_bb3f_ea72ef3060da.slice/crio-f1d454e82bc4397d0f9d42b77ac248bf16454f44f91ab23f90aa7d5fc326ad3e WatchSource:0}: Error finding container f1d454e82bc4397d0f9d42b77ac248bf16454f44f91ab23f90aa7d5fc326ad3e: Status 404 returned error can't find the container with id f1d454e82bc4397d0f9d42b77ac248bf16454f44f91ab23f90aa7d5fc326ad3e Oct 30 19:09:22.768427 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.768271 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 19:09:22.779677 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.779637 4582 generic.go:356] "Generic (PLEG): container finished" podID="cad19fb9-c8d7-48f0-9d05-992cafb88bbc" containerID="61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69" exitCode=0 Oct 30 19:09:22.779864 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.779676 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" Oct 30 19:09:22.779864 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.779720 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" event={"ID":"cad19fb9-c8d7-48f0-9d05-992cafb88bbc","Type":"ContainerDied","Data":"61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69"} Oct 30 19:09:22.779864 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.779811 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc" event={"ID":"cad19fb9-c8d7-48f0-9d05-992cafb88bbc","Type":"ContainerDied","Data":"c72ddd10e5acbbd99a514e7e8f03af28256ec79387c0d2d1d2ecf1f0c0ed2755"} Oct 30 19:09:22.779864 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.779858 4582 scope.go:122] "RemoveContainer" containerID="61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69" Oct 30 19:09:22.781386 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.781360 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" event={"ID":"7835c137-0861-48e2-bb3f-ea72ef3060da","Type":"ContainerStarted","Data":"f1d454e82bc4397d0f9d42b77ac248bf16454f44f91ab23f90aa7d5fc326ad3e"} Oct 30 19:09:22.791899 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.791882 4582 scope.go:122] "RemoveContainer" containerID="61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69" Oct 30 19:09:22.792182 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:09:22.792161 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69\": container with ID starting with 61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69 not found: ID does not exist" containerID="61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69" Oct 30 19:09:22.792251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.792189 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69"} err="failed to get container status \"61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69\": rpc error: code = NotFound desc = could not find container \"61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69\": container with ID starting with 61aa7e14e2c00967a9a513d1e4892e99d27e959633a1d86cba479aa95d681d69 not found: ID does not exist" Oct 30 19:09:22.883313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.883282 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-maas-api-tls\") pod \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " Oct 30 19:09:22.883506 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.883374 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-openshift-service-ca\") pod \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " Oct 30 19:09:22.883506 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.883442 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-metrics-certs\") pod \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " Oct 30 19:09:22.883506 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.883500 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-kube-api-access-2whff\" (UniqueName: \"kubernetes.io/projected/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-kube-api-access-2whff\") pod \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\" (UID: \"cad19fb9-c8d7-48f0-9d05-992cafb88bbc\") " Oct 30 19:09:22.883835 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.883756 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-openshift-service-ca" pod "cad19fb9-c8d7-48f0-9d05-992cafb88bbc" (UID: "cad19fb9-c8d7-48f0-9d05-992cafb88bbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:22.884127 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.884103 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:22.885643 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.885619 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-metrics-certs" pod "cad19fb9-c8d7-48f0-9d05-992cafb88bbc" (UID: "cad19fb9-c8d7-48f0-9d05-992cafb88bbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:22.885804 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.885787 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-kube-api-access-2whff" pod "cad19fb9-c8d7-48f0-9d05-992cafb88bbc" (UID: "cad19fb9-c8d7-48f0-9d05-992cafb88bbc"). InnerVolumeSpecName "kube-api-access-2whff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:22.885973 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.885950 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-maas-api-tls" pod "cad19fb9-c8d7-48f0-9d05-992cafb88bbc" (UID: "cad19fb9-c8d7-48f0-9d05-992cafb88bbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:22.985149 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.985113 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:22.985149 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.985143 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whff\" (UniqueName: \"kubernetes.io/projected/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-kube-api-access-2whff\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:22.985378 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:22.985162 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cad19fb9-c8d7-48f0-9d05-992cafb88bbc-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:23.125107 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.125070 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc"] Oct 30 19:09:23.136999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.136930 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ab244c-789d8b79bd-ftfbc"] Oct 30 19:09:23.303158 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.303125 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad19fb9-c8d7-48f0-9d05-992cafb88bbc" path="/var/lib/kubelet/pods/cad19fb9-c8d7-48f0-9d05-992cafb88bbc/volumes" Oct 30 19:09:23.309281 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.309250 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:23.452385 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.452301 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:23.791319 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.791225 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" event={"ID":"7835c137-0861-48e2-bb3f-ea72ef3060da","Type":"ContainerStarted","Data":"2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9"} Oct 30 19:09:23.833039 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.832992 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" podStartSLOduration=1.832978928 podStartE2EDuration="1.832978928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:23.831317993 +0000 UTC m=+4260.655712285" watchObservedRunningTime="2026-10-30 19:09:23.832978928 +0000 UTC m=+4260.657373220" Oct 30 19:09:23.905290 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:23.905253 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:24.364127 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:24.364097 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:25.404060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:25.404026 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:27.064749 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:27.064720 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:28.714337 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:28.714278 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:30.590400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:30.590362 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:30.727396 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:30.727355 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:30.830756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:30.830687 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:31.077980 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:31.077948 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:32.363632 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:32.363602 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:32.375765 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:32.375735 4582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:32.600736 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:32.600702 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:32.863513 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:32.863484 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:09:34.491781 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:34.491747 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:35.866312 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:35.866271 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8"] Oct 30 19:09:35.866848 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:35.866519 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" podUID="96bab029-b5b1-40d4-bb1d-df4a3461ee67" containerName="maas-api" containerID="cri-o://cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541" gracePeriod=30 Oct 30 19:09:35.918476 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:35.918447 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:36.153551 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.153522 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 19:09:36.234293 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:09:36.234240 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:09:36.247073 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.247044 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bab029-b5b1-40d4-bb1d-df4a3461ee67-kube-api-access-dvd4c\" (UniqueName: \"kubernetes.io/projected/96bab029-b5b1-40d4-bb1d-df4a3461ee67-kube-api-access-dvd4c\") pod \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " Oct 30 19:09:36.247254 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.247168 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-maas-api-tls\") pod \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " Oct 30 19:09:36.247374 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.247356 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-metrics-certs\") pod \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " Oct 30 19:09:36.247467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.247391 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96bab029-b5b1-40d4-bb1d-df4a3461ee67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bab029-b5b1-40d4-bb1d-df4a3461ee67-openshift-service-ca\") pod \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\" (UID: \"96bab029-b5b1-40d4-bb1d-df4a3461ee67\") " Oct 30 19:09:36.247895 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.247857 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bab029-b5b1-40d4-bb1d-df4a3461ee67-openshift-service-ca" pod "96bab029-b5b1-40d4-bb1d-df4a3461ee67" (UID: "96bab029-b5b1-40d4-bb1d-df4a3461ee67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:36.248035 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.248003 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bab029-b5b1-40d4-bb1d-df4a3461ee67-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.249375 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.249349 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bab029-b5b1-40d4-bb1d-df4a3461ee67-kube-api-access-dvd4c" pod "96bab029-b5b1-40d4-bb1d-df4a3461ee67" (UID: "96bab029-b5b1-40d4-bb1d-df4a3461ee67"). InnerVolumeSpecName "kube-api-access-dvd4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:36.249476 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.249430 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-metrics-certs" pod "96bab029-b5b1-40d4-bb1d-df4a3461ee67" (UID: "96bab029-b5b1-40d4-bb1d-df4a3461ee67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:36.249476 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.249459 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-maas-api-tls" pod "96bab029-b5b1-40d4-bb1d-df4a3461ee67" (UID: "96bab029-b5b1-40d4-bb1d-df4a3461ee67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:36.349816 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.349776 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.349816 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.349812 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvd4c\" (UniqueName: \"kubernetes.io/projected/96bab029-b5b1-40d4-bb1d-df4a3461ee67-kube-api-access-dvd4c\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.350113 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.349838 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bab029-b5b1-40d4-bb1d-df4a3461ee67-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.648227 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.648187 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:36.756846 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.756784 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:36.863656 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.863613 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:36.882377 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.882344 4582 generic.go:356] "Generic (PLEG): container finished" podID="96bab029-b5b1-40d4-bb1d-df4a3461ee67" containerID="cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541" exitCode=0 Oct 30 19:09:36.882896 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.882400 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" Oct 30 19:09:36.882896 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.882427 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" event={"ID":"96bab029-b5b1-40d4-bb1d-df4a3461ee67","Type":"ContainerDied","Data":"cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541"} Oct 30 19:09:36.882896 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.882519 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8" event={"ID":"96bab029-b5b1-40d4-bb1d-df4a3461ee67","Type":"ContainerDied","Data":"0beb1ff5603c391df3566ef7f862b31e557e952402a32d3cf57d7acae6d51f22"} Oct 30 19:09:36.882896 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.882555 4582 scope.go:122] "RemoveContainer" containerID="cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541" Oct 30 19:09:36.893522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.893506 4582 scope.go:122] "RemoveContainer" containerID="cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541" Oct 30 19:09:36.893784 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:09:36.893760 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541\": container with ID starting with cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541 not found: ID does not exist" containerID="cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541" Oct 30 19:09:36.893871 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.893796 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541"} err="failed to get container status \"cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541\": rpc error: code = NotFound desc = could not find container \"cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541\": container with ID starting with cf190c500a4e7693a975c38a6676b2700c294bac30a80aaca88f2cf947f91541 not found: ID does not exist" Oct 30 19:09:36.933453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.933386 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8"] Oct 30 19:09:36.944999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:36.944974 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d82e82-cb9777bd4-rhhv8"] Oct 30 19:09:37.287265 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:37.287185 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:37.301337 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:37.301301 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bab029-b5b1-40d4-bb1d-df4a3461ee67" path="/var/lib/kubelet/pods/96bab029-b5b1-40d4-bb1d-df4a3461ee67/volumes" Oct 30 19:09:38.623157 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:38.623111 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:40.467985 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:40.467944 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:42.117815 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:42.117744 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq"] Oct 30 19:09:42.168992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:42.168954 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4"] Oct 30 19:09:42.350811 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:42.350779 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:42.463342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:42.463265 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:42.625341 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:42.625308 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:44.356096 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:44.356062 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:44.490485 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:44.490449 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:44.724606 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:44.724571 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:44.837429 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:44.837388 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:45.164645 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:45.164609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:46.675367 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:46.675330 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:47.118898 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:47.118854 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" podUID="1f410e59-f1d4-4741-9d50-ba48d7b67588" containerName="payload-processing" containerID="cri-o://9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec" gracePeriod=25 Oct 30 19:09:47.170100 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:47.170059 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" podUID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" containerName="payload-pre-processing" containerID="cri-o://7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5" gracePeriod=25 Oct 30 19:09:48.591285 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:48.591249 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:49.505589 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:49.505556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:50.123066 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:50.123033 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d"] Oct 30 19:09:50.133869 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:50.133835 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh"] Oct 30 19:09:50.225207 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:50.225172 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:50.469776 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:50.469699 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:52.161108 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:52.161068 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:52.487222 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:52.487128 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" podUID="1f410e59-f1d4-4741-9d50-ba48d7b67588" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:52.487222 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:52.487165 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" podUID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:09:53.877837 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:53.877794 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:55.568620 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:55.568584 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:56.020841 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:56.020276 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:56.116889 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:56.116846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:56.887635 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:56.887604 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:57.067676 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.067633 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv"] Oct 30 19:09:57.067959 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.067873 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" podUID="9997e625-9d15-4297-8813-cbde3f862c8b" containerName="maas-api" containerID="cri-o://1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f" gracePeriod=30 Oct 30 19:09:57.117523 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.117495 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:57.329261 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.329232 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 19:09:57.520807 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.520730 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9997e625-9d15-4297-8813-cbde3f862c8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9997e625-9d15-4297-8813-cbde3f862c8b-openshift-service-ca\") pod \"9997e625-9d15-4297-8813-cbde3f862c8b\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " Oct 30 19:09:57.520990 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.520881 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-metrics-certs\") pod \"9997e625-9d15-4297-8813-cbde3f862c8b\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " Oct 30 19:09:57.520990 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.520920 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-maas-api-tls\") pod \"9997e625-9d15-4297-8813-cbde3f862c8b\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " Oct 30 19:09:57.521081 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.520992 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9997e625-9d15-4297-8813-cbde3f862c8b-kube-api-access-4cd7r\" (UniqueName: \"kubernetes.io/projected/9997e625-9d15-4297-8813-cbde3f862c8b-kube-api-access-4cd7r\") pod \"9997e625-9d15-4297-8813-cbde3f862c8b\" (UID: \"9997e625-9d15-4297-8813-cbde3f862c8b\") " Oct 30 19:09:57.521199 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.521171 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9997e625-9d15-4297-8813-cbde3f862c8b-openshift-service-ca" pod "9997e625-9d15-4297-8813-cbde3f862c8b" (UID: "9997e625-9d15-4297-8813-cbde3f862c8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:57.521622 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.521575 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9997e625-9d15-4297-8813-cbde3f862c8b-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:57.523153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.523125 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-metrics-certs" pod "9997e625-9d15-4297-8813-cbde3f862c8b" (UID: "9997e625-9d15-4297-8813-cbde3f862c8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:57.523266 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.523227 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-maas-api-tls" pod "9997e625-9d15-4297-8813-cbde3f862c8b" (UID: "9997e625-9d15-4297-8813-cbde3f862c8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:57.523375 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.523359 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9997e625-9d15-4297-8813-cbde3f862c8b-kube-api-access-4cd7r" pod "9997e625-9d15-4297-8813-cbde3f862c8b" (UID: "9997e625-9d15-4297-8813-cbde3f862c8b"). InnerVolumeSpecName "kube-api-access-4cd7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:57.622840 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.622801 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:57.623022 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.622849 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cd7r\" (UniqueName: \"kubernetes.io/projected/9997e625-9d15-4297-8813-cbde3f862c8b-kube-api-access-4cd7r\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:57.623022 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.622861 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9997e625-9d15-4297-8813-cbde3f862c8b-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:09:57.840663 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.840586 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:57.968711 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:57.968681 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:58.038763 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.038729 4582 generic.go:356] "Generic (PLEG): container finished" podID="9997e625-9d15-4297-8813-cbde3f862c8b" containerID="1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f" exitCode=0 Oct 30 19:09:58.038992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.038844 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" Oct 30 19:09:58.038992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.038845 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" event={"ID":"9997e625-9d15-4297-8813-cbde3f862c8b","Type":"ContainerDied","Data":"1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f"} Oct 30 19:09:58.038992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.038930 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv" event={"ID":"9997e625-9d15-4297-8813-cbde3f862c8b","Type":"ContainerDied","Data":"1e6dce0a828905216762776a38ba362f44b8189dc7b7174fe491d8db61660793"} Oct 30 19:09:58.038992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.038968 4582 scope.go:122] "RemoveContainer" containerID="1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f" Oct 30 19:09:58.049692 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.049676 4582 scope.go:122] "RemoveContainer" containerID="1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f" Oct 30 19:09:58.049983 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:09:58.049965 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f\": container with ID starting with 1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f not found: ID does not exist" containerID="1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f" Oct 30 19:09:58.050057 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.049988 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f"} err="failed to get container status \"1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f\": rpc error: code = NotFound desc = could not find container \"1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f\": container with ID starting with 1b7ecb15cbe2cbb451f36348b485a4ff1f0133d1080a04afa13628ac0c59e71f not found: ID does not exist" Oct 30 19:09:58.071343 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.071319 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:58.097012 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.096952 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv"] Oct 30 19:09:58.108582 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.108561 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a7508-c5cfd7958-n8xxv"] Oct 30 19:09:58.462358 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:58.462329 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:59.301272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:59.301243 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9997e625-9d15-4297-8813-cbde3f862c8b" path="/var/lib/kubelet/pods/9997e625-9d15-4297-8813-cbde3f862c8b/volumes" Oct 30 19:09:59.993768 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:09:59.993732 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:01.769893 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:01.769858 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:02.487738 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:02.487695 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" podUID="1f410e59-f1d4-4741-9d50-ba48d7b67588" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:02.487958 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:02.487767 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" podUID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:10:03.572497 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:03.572455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:05.144998 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.144956 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" containerID="cri-o://2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a" gracePeriod=45 Oct 30 19:10:05.155129 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.155097 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:05.157234 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.157207 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" containerID="cri-o://2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77" gracePeriod=45 Oct 30 19:10:05.202152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.202116 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused" Oct 30 19:10:05.294758 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.294717 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:05.458720 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.458690 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 19:10:05.462308 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.462286 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 19:10:05.516760 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.516730 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tmp-dir\") pod \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " Oct 30 19:10:05.516976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.516786 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-home\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-home\") pod \"cf04596b-f187-4b45-b528-2e6698a13711\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " Oct 30 19:10:05.516976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.516808 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-dshm\") pod \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " Oct 30 19:10:05.516976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.516869 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3260d1a7-a604-4bc2-b904-a2fe7129bd23-kube-api-access-9ml4w\" (UniqueName: \"kubernetes.io/projected/3260d1a7-a604-4bc2-b904-a2fe7129bd23-kube-api-access-9ml4w\") pod \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " Oct 30 19:10:05.516976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.516910 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-dshm\") pod \"cf04596b-f187-4b45-b528-2e6698a13711\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " Oct 30 19:10:05.516976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.516945 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-model-cache\") pod \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " Oct 30 19:10:05.517273 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517014 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-home\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-home\") pod \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " Oct 30 19:10:05.517273 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517035 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tmp-dir" pod "3260d1a7-a604-4bc2-b904-a2fe7129bd23" (UID: "3260d1a7-a604-4bc2-b904-a2fe7129bd23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.517273 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517095 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-home" pod "cf04596b-f187-4b45-b528-2e6698a13711" (UID: "cf04596b-f187-4b45-b528-2e6698a13711"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.517498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517272 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-model-cache" pod "cf04596b-f187-4b45-b528-2e6698a13711" (UID: "cf04596b-f187-4b45-b528-2e6698a13711"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.517498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517292 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-model-cache" pod "3260d1a7-a604-4bc2-b904-a2fe7129bd23" (UID: "3260d1a7-a604-4bc2-b904-a2fe7129bd23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.517498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517047 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-model-cache\") pod \"cf04596b-f187-4b45-b528-2e6698a13711\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " Oct 30 19:10:05.517498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517422 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tls-certs\" (UniqueName: \"kubernetes.io/secret/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tls-certs\") pod \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\" (UID: \"3260d1a7-a604-4bc2-b904-a2fe7129bd23\") " Oct 30 19:10:05.517719 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517498 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-tmp-dir\") pod \"cf04596b-f187-4b45-b528-2e6698a13711\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " Oct 30 19:10:05.517719 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517527 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf04596b-f187-4b45-b528-2e6698a13711-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf04596b-f187-4b45-b528-2e6698a13711-tls-certs\") pod \"cf04596b-f187-4b45-b528-2e6698a13711\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " Oct 30 19:10:05.517719 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517622 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf04596b-f187-4b45-b528-2e6698a13711-kube-api-access-8l7s7\" (UniqueName: \"kubernetes.io/projected/cf04596b-f187-4b45-b528-2e6698a13711-kube-api-access-8l7s7\") pod \"cf04596b-f187-4b45-b528-2e6698a13711\" (UID: \"cf04596b-f187-4b45-b528-2e6698a13711\") " Oct 30 19:10:05.517909 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.517409 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-home" pod "3260d1a7-a604-4bc2-b904-a2fe7129bd23" (UID: "3260d1a7-a604-4bc2-b904-a2fe7129bd23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.518387 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.518066 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-tmp-dir" pod "cf04596b-f187-4b45-b528-2e6698a13711" (UID: "cf04596b-f187-4b45-b528-2e6698a13711"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.519040 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.519017 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.519859 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.519333 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.520001 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.519988 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.520107 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.520094 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.520210 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.520200 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.520322 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.520302 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.520552 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.520520 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-dshm" pod "3260d1a7-a604-4bc2-b904-a2fe7129bd23" (UID: "3260d1a7-a604-4bc2-b904-a2fe7129bd23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.520663 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.520583 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf04596b-f187-4b45-b528-2e6698a13711-kube-api-access-8l7s7" pod "cf04596b-f187-4b45-b528-2e6698a13711" (UID: "cf04596b-f187-4b45-b528-2e6698a13711"). InnerVolumeSpecName "kube-api-access-8l7s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:05.521163 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.521126 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3260d1a7-a604-4bc2-b904-a2fe7129bd23-kube-api-access-9ml4w" pod "3260d1a7-a604-4bc2-b904-a2fe7129bd23" (UID: "3260d1a7-a604-4bc2-b904-a2fe7129bd23"). InnerVolumeSpecName "kube-api-access-9ml4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:05.521388 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.521360 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-dshm" pod "cf04596b-f187-4b45-b528-2e6698a13711" (UID: "cf04596b-f187-4b45-b528-2e6698a13711"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.521476 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.521416 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tls-certs" pod "3260d1a7-a604-4bc2-b904-a2fe7129bd23" (UID: "3260d1a7-a604-4bc2-b904-a2fe7129bd23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:05.523152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.523121 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf04596b-f187-4b45-b528-2e6698a13711-tls-certs" pod "cf04596b-f187-4b45-b528-2e6698a13711" (UID: "cf04596b-f187-4b45-b528-2e6698a13711"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:05.567077 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.567043 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:05.621751 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.621712 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3260d1a7-a604-4bc2-b904-a2fe7129bd23-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.621751 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.621753 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ml4w\" (UniqueName: \"kubernetes.io/projected/3260d1a7-a604-4bc2-b904-a2fe7129bd23-kube-api-access-9ml4w\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.621977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.621769 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf04596b-f187-4b45-b528-2e6698a13711-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.621977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.621783 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3260d1a7-a604-4bc2-b904-a2fe7129bd23-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.621977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.621796 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf04596b-f187-4b45-b528-2e6698a13711-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.621977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.621809 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l7s7\" (UniqueName: \"kubernetes.io/projected/cf04596b-f187-4b45-b528-2e6698a13711-kube-api-access-8l7s7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.882570 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.882537 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:05.988474 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:05.988441 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:06.104323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.104284 4582 generic.go:356] "Generic (PLEG): container finished" podID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerID="2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77" exitCode=0 Oct 30 19:10:06.104528 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.104333 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" Oct 30 19:10:06.104528 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.104370 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" event={"ID":"3260d1a7-a604-4bc2-b904-a2fe7129bd23","Type":"ContainerDied","Data":"2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77"} Oct 30 19:10:06.104528 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.104420 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh" event={"ID":"3260d1a7-a604-4bc2-b904-a2fe7129bd23","Type":"ContainerDied","Data":"177dcaa631c7b3bbcc669aa32646beb3b8aa48b4a9938543c0785b8dbc3e389d"} Oct 30 19:10:06.104528 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.104444 4582 scope.go:122] "RemoveContainer" containerID="2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77" Oct 30 19:10:06.106202 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.106176 4582 generic.go:356] "Generic (PLEG): container finished" podID="cf04596b-f187-4b45-b528-2e6698a13711" containerID="2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a" exitCode=0 Oct 30 19:10:06.106323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.106214 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" event={"ID":"cf04596b-f187-4b45-b528-2e6698a13711","Type":"ContainerDied","Data":"2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a"} Oct 30 19:10:06.106323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.106225 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" Oct 30 19:10:06.106323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.106241 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d" event={"ID":"cf04596b-f187-4b45-b528-2e6698a13711","Type":"ContainerDied","Data":"7ac269dab4d6c4c5c6f3fc8febbb8c0851cfa674bc2834c7e9133343a57ab7d7"} Oct 30 19:10:06.117723 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.117705 4582 scope.go:122] "RemoveContainer" containerID="2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77" Oct 30 19:10:06.118192 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:10:06.118035 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77\": container with ID starting with 2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77 not found: ID does not exist" containerID="2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77" Oct 30 19:10:06.118192 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.118062 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77"} err="failed to get container status \"2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77\": rpc error: code = NotFound desc = could not find container \"2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77\": container with ID starting with 2e9e5a53f97e403ad2121ca27b73ba5f68ba8e3f19a3975692d09c838bf76d77 not found: ID does not exist" Oct 30 19:10:06.118192 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.118080 4582 scope.go:122] "RemoveContainer" containerID="2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a" Oct 30 19:10:06.127445 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.127424 4582 scope.go:122] "RemoveContainer" containerID="2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a" Oct 30 19:10:06.127719 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:10:06.127696 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a\": container with ID starting with 2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a not found: ID does not exist" containerID="2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a" Oct 30 19:10:06.127795 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.127729 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a"} err="failed to get container status \"2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a\": rpc error: code = NotFound desc = could not find container \"2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a\": container with ID starting with 2011ddd432201a95ef16b243050eed66ef1a965a6553c9c553e00bad8fcc335a not found: ID does not exist" Oct 30 19:10:06.157269 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.157194 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh"] Oct 30 19:10:06.168507 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.168485 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-w1-380c08-kserve-66ff8cc4-8f5bh"] Oct 30 19:10:06.200778 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.200747 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d"] Oct 30 19:10:06.209522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:06.209490 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/distinct-2-w1-380c08-kserve-6fc7599fbf-gd48d"] Oct 30 19:10:06.238117 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:10:06.238080 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:10:07.306850 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.306800 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" path="/var/lib/kubelet/pods/3260d1a7-a604-4bc2-b904-a2fe7129bd23/volumes" Oct 30 19:10:07.307571 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.307547 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf04596b-f187-4b45-b528-2e6698a13711" path="/var/lib/kubelet/pods/cf04596b-f187-4b45-b528-2e6698a13711/volumes" Oct 30 19:10:07.443043 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.443019 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 19:10:07.477038 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.477015 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 19:10:07.550872 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.550840 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-kube-api-access-t7ttw\" (UniqueName: \"kubernetes.io/projected/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-kube-api-access-t7ttw\") pod \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " Oct 30 19:10:07.551026 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.550945 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f410e59-f1d4-4741-9d50-ba48d7b67588-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f410e59-f1d4-4741-9d50-ba48d7b67588-plugins-config-volume\") pod \"1f410e59-f1d4-4741-9d50-ba48d7b67588\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " Oct 30 19:10:07.551026 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.550988 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-plugins-config-volume\") pod \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\" (UID: \"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e\") " Oct 30 19:10:07.551169 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.551076 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f410e59-f1d4-4741-9d50-ba48d7b67588-kube-api-access-zdjdl\" (UniqueName: \"kubernetes.io/projected/1f410e59-f1d4-4741-9d50-ba48d7b67588-kube-api-access-zdjdl\") pod \"1f410e59-f1d4-4741-9d50-ba48d7b67588\" (UID: \"1f410e59-f1d4-4741-9d50-ba48d7b67588\") " Oct 30 19:10:07.551439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.551408 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f410e59-f1d4-4741-9d50-ba48d7b67588-plugins-config-volume" pod "1f410e59-f1d4-4741-9d50-ba48d7b67588" (UID: "1f410e59-f1d4-4741-9d50-ba48d7b67588"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:07.551439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.551422 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-plugins-config-volume" pod "9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" (UID: "9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:07.551737 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.551704 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f410e59-f1d4-4741-9d50-ba48d7b67588-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:07.551737 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.551728 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:07.553294 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.553267 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-kube-api-access-t7ttw" pod "9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" (UID: "9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e"). InnerVolumeSpecName "kube-api-access-t7ttw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:07.553378 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.553306 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f410e59-f1d4-4741-9d50-ba48d7b67588-kube-api-access-zdjdl" pod "1f410e59-f1d4-4741-9d50-ba48d7b67588" (UID: "1f410e59-f1d4-4741-9d50-ba48d7b67588"). InnerVolumeSpecName "kube-api-access-zdjdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:07.653253 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.653183 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdjdl\" (UniqueName: \"kubernetes.io/projected/1f410e59-f1d4-4741-9d50-ba48d7b67588-kube-api-access-zdjdl\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:07.653253 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.653208 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ttw\" (UniqueName: \"kubernetes.io/projected/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e-kube-api-access-t7ttw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:07.873713 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:07.873681 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:08.123578 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.123546 4582 generic.go:356] "Generic (PLEG): container finished" podID="1f410e59-f1d4-4741-9d50-ba48d7b67588" containerID="9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec" exitCode=0 Oct 30 19:10:08.123764 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.123595 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" Oct 30 19:10:08.123764 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.123626 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" event={"ID":"1f410e59-f1d4-4741-9d50-ba48d7b67588","Type":"ContainerDied","Data":"9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec"} Oct 30 19:10:08.123764 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.123697 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq" event={"ID":"1f410e59-f1d4-4741-9d50-ba48d7b67588","Type":"ContainerDied","Data":"dc3fedf8b1904cb477b2449c853a5133090d68b179a0160b0c06fb5243aa5dc5"} Oct 30 19:10:08.123764 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.123724 4582 scope.go:122] "RemoveContainer" containerID="9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec" Oct 30 19:10:08.125460 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.125427 4582 generic.go:356] "Generic (PLEG): container finished" podID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" containerID="7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5" exitCode=0 Oct 30 19:10:08.125586 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.125477 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" Oct 30 19:10:08.125586 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.125483 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" event={"ID":"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e","Type":"ContainerDied","Data":"7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5"} Oct 30 19:10:08.125586 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.125534 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4" event={"ID":"9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e","Type":"ContainerDied","Data":"805d075b433e3da9ea70c4715fc2b7802a85630c0bb80c12a5c396edc0b7fce8"} Oct 30 19:10:08.134468 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.134448 4582 scope.go:122] "RemoveContainer" containerID="9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec" Oct 30 19:10:08.134766 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:10:08.134748 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec\": container with ID starting with 9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec not found: ID does not exist" containerID="9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec" Oct 30 19:10:08.134851 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.134774 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec"} err="failed to get container status \"9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec\": rpc error: code = NotFound desc = could not find container \"9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec\": container with ID starting with 9de74221d4cb2b3c6541852f1d6a1ad8d190ad53fc733718b5113db8a3cb86ec not found: ID does not exist" Oct 30 19:10:08.134851 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.134791 4582 scope.go:122] "RemoveContainer" containerID="7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5" Oct 30 19:10:08.144061 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.144038 4582 scope.go:122] "RemoveContainer" containerID="7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5" Oct 30 19:10:08.144342 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:10:08.144324 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5\": container with ID starting with 7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5 not found: ID does not exist" containerID="7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5" Oct 30 19:10:08.144417 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.144352 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5"} err="failed to get container status \"7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5\": rpc error: code = NotFound desc = could not find container \"7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5\": container with ID starting with 7e86cab8c2e71acd00126099427173856bc842f4b9452b9db71a0f3b446b7fb5 not found: ID does not exist" Oct 30 19:10:08.174042 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.174015 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4"] Oct 30 19:10:08.185808 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.185780 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4a7508-78598dfc9f-wk8g4"] Oct 30 19:10:08.218334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.218303 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq"] Oct 30 19:10:08.221062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:08.221040 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4a7508-6f5b58db4b-skjjq"] Oct 30 19:10:09.301379 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:09.301347 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f410e59-f1d4-4741-9d50-ba48d7b67588" path="/var/lib/kubelet/pods/1f410e59-f1d4-4741-9d50-ba48d7b67588/volumes" Oct 30 19:10:09.301861 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:09.301846 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" path="/var/lib/kubelet/pods/9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e/volumes" Oct 30 19:10:09.563224 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:09.563144 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:10.944362 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:10.944332 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv"] Oct 30 19:10:11.019258 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:11.019224 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:11.264323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:11.264247 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:13.071806 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:13.071767 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:14.829381 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:14.829348 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:16.655637 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:16.655599 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:16.848797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:16.848751 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:16.956574 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:16.956456 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:18.474716 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:18.474668 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:19.958644 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:19.958609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:20.169867 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:20.169792 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:20.270885 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:20.270759 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:21.871389 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:21.871330 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:23.643815 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:23.643780 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:25.351964 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:25.351919 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:25.966008 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:25.965970 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" containerID="cri-o://7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c" gracePeriod=45 Oct 30 19:10:26.223659 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.223596 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 19:10:26.244370 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.244340 4582 generic.go:356] "Generic (PLEG): container finished" podID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerID="7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c" exitCode=0 Oct 30 19:10:26.244531 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.244383 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" Oct 30 19:10:26.244531 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.244428 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" event={"ID":"feccd1bd-7683-45ee-bbe1-4d20d06491d1","Type":"ContainerDied","Data":"7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c"} Oct 30 19:10:26.244531 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.244482 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv" event={"ID":"feccd1bd-7683-45ee-bbe1-4d20d06491d1","Type":"ContainerDied","Data":"74afbd9a666f78ba6b6856374c70872d579ee9634fce67a20593e19b51eb5429"} Oct 30 19:10:26.244531 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.244507 4582 scope.go:122] "RemoveContainer" containerID="7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c" Oct 30 19:10:26.253780 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.253764 4582 scope.go:122] "RemoveContainer" containerID="7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c" Oct 30 19:10:26.254043 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:10:26.254026 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c\": container with ID starting with 7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c not found: ID does not exist" containerID="7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c" Oct 30 19:10:26.254108 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.254050 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c"} err="failed to get container status \"7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c\": rpc error: code = NotFound desc = could not find container \"7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c\": container with ID starting with 7e20c529f663300675e9f5699805f42f9bc1f2ef0c954070f972e9a739f2c06c not found: ID does not exist" Oct 30 19:10:26.277075 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277037 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-model-cache\") pod \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " Oct 30 19:10:26.277215 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277112 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tls-certs\") pod \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " Oct 30 19:10:26.277295 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277281 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tmp-dir\") pod \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " Oct 30 19:10:26.277357 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277336 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feccd1bd-7683-45ee-bbe1-4d20d06491d1-kube-api-access-fbwbm\" (UniqueName: \"kubernetes.io/projected/feccd1bd-7683-45ee-bbe1-4d20d06491d1-kube-api-access-fbwbm\") pod \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " Oct 30 19:10:26.277420 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277367 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-home\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-home\") pod \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " Oct 30 19:10:26.277420 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277394 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-dshm\") pod \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\" (UID: \"feccd1bd-7683-45ee-bbe1-4d20d06491d1\") " Oct 30 19:10:26.277560 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277537 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tmp-dir" pod "feccd1bd-7683-45ee-bbe1-4d20d06491d1" (UID: "feccd1bd-7683-45ee-bbe1-4d20d06491d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.277976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277938 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-home" pod "feccd1bd-7683-45ee-bbe1-4d20d06491d1" (UID: "feccd1bd-7683-45ee-bbe1-4d20d06491d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.277976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277972 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.278272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.277379 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-model-cache" pod "feccd1bd-7683-45ee-bbe1-4d20d06491d1" (UID: "feccd1bd-7683-45ee-bbe1-4d20d06491d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.280953 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.280927 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tls-certs" pod "feccd1bd-7683-45ee-bbe1-4d20d06491d1" (UID: "feccd1bd-7683-45ee-bbe1-4d20d06491d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:26.280953 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.280943 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-dshm" pod "feccd1bd-7683-45ee-bbe1-4d20d06491d1" (UID: "feccd1bd-7683-45ee-bbe1-4d20d06491d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.281134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.280956 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feccd1bd-7683-45ee-bbe1-4d20d06491d1-kube-api-access-fbwbm" pod "feccd1bd-7683-45ee-bbe1-4d20d06491d1" (UID: "feccd1bd-7683-45ee-bbe1-4d20d06491d1"). InnerVolumeSpecName "kube-api-access-fbwbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:26.379467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.379434 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbwbm\" (UniqueName: \"kubernetes.io/projected/feccd1bd-7683-45ee-bbe1-4d20d06491d1-kube-api-access-fbwbm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.379467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.379465 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.379467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.379475 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.379999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.379485 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feccd1bd-7683-45ee-bbe1-4d20d06491d1-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.379999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.379495 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feccd1bd-7683-45ee-bbe1-4d20d06491d1-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.591405 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.591374 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv"] Oct 30 19:10:26.603201 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:26.603172 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/premium-w3-de7e3a-kserve-7d48c85ff4-sl6dv"] Oct 30 19:10:27.012122 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:27.012089 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:27.112263 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:27.112231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:27.301111 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:27.301024 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" path="/var/lib/kubelet/pods/feccd1bd-7683-45ee-bbe1-4d20d06491d1/volumes" Oct 30 19:10:27.313738 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:27.313708 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:27.504293 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:27.504256 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:27.610798 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:27.610715 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:28.425454 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:28.425422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:28.967512 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:28.967478 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:30.677788 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:30.677754 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:32.542568 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:32.542537 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:34.170345 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:34.170310 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:34.503498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:34.503416 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:34.602767 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:34.602731 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:35.718030 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:35.717991 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:36.228790 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:10:36.228741 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:10:36.330460 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:36.330415 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:36.504756 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:36.504670 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:36.704482 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:36.704447 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:37.008132 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:37.008073 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:38.575599 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:38.575562 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:40.496262 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:40.496231 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.317554 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:42.317518 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:44.043973 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:44.043923 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:44.144104 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:44.144043 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:44.424472 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:44.424431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:45.778752 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:45.778716 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:47.715808 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:47.715776 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:48.147509 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:48.147469 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:48.259848 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:48.259788 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:49.647279 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:49.647244 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:49.904122 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:49.904037 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:51.418989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:51.418956 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.290427 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:53.290398 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:54.974656 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:54.974622 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:56.288565 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:56.288529 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:56.411871 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:56.411809 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:56.835778 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:56.835735 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:58.515666 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:10:58.515625 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:00.199632 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:00.199587 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:02.080910 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:02.080869 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:03.758866 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:03.758816 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:05.355050 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:05.355015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:06.003510 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:06.003479 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:06.107709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:06.107676 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:06.212400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:06.212368 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:06.228307 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:11:06.228271 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:11:07.894181 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:07.894140 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:09.759069 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:09.759030 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:11.511239 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:11.511206 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:13.181578 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:13.181548 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:14.875055 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:14.875019 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:16.669425 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:16.669394 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:18.371800 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:18.371757 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:20.083865 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:20.083811 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:20.184464 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:20.184417 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:21.968393 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:21.968358 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:23.775899 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:23.775860 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:25.520672 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:25.520627 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:26.616343 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:26.616311 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:27.187445 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:27.187409 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:28.899217 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:28.899184 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:30.663587 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:30.663549 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:32.395260 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:32.395226 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:32.498035 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:32.497999 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:32.779631 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:32.779542 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr"] Oct 30 19:11:32.779889 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:32.779839 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" containerID="cri-o://b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0" gracePeriod=30 Oct 30 19:11:32.806285 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:32.806257 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:34.158653 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:34.158620 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:35.830524 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:35.830491 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:36.229920 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:11:36.229880 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:11:37.762906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:37.762873 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:38.443786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.443760 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 19:11:38.524802 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.524771 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-socket\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525003 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.524845 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-envoy\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525003 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.524875 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-token\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-token\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525003 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.524977 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-credential-socket\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525008 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-certs\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525065 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istiod-ca-cert\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525104 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-kube-api-access-jm68k\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-kube-api-access-jm68k\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525126 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-socket" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:38.525167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525140 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-data\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525446 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525167 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-podinfo\") pod \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\" (UID: \"fa4187a2-d01f-487d-8ebe-5c835efc4eee\") " Oct 30 19:11:38.525446 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525327 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-credential-socket" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:38.525563 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525482 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istiod-ca-cert" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:38.525622 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525567 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-data" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:38.525961 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525928 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-certs" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:38.525961 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525945 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.526139 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525964 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.526139 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525980 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.526139 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.525992 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.527949 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.527916 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-token" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:38.528076 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.528026 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-envoy" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:38.528076 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.528035 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-podinfo" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:11:38.531086 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.531060 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-kube-api-access-jm68k" pod "fa4187a2-d01f-487d-8ebe-5c835efc4eee" (UID: "fa4187a2-d01f-487d-8ebe-5c835efc4eee"). InnerVolumeSpecName "kube-api-access-jm68k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:38.627219 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.627185 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.627219 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.627215 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.627219 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.627225 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa4187a2-d01f-487d-8ebe-5c835efc4eee-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.627440 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.627234 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm68k\" (UniqueName: \"kubernetes.io/projected/fa4187a2-d01f-487d-8ebe-5c835efc4eee-kube-api-access-jm68k\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.627440 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.627244 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa4187a2-d01f-487d-8ebe-5c835efc4eee-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:11:38.686935 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.686901 4582 generic.go:356] "Generic (PLEG): container finished" podID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerID="b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0" exitCode=0 Oct 30 19:11:38.687111 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.686952 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" Oct 30 19:11:38.687111 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.686978 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" event={"ID":"fa4187a2-d01f-487d-8ebe-5c835efc4eee","Type":"ContainerDied","Data":"b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0"} Oct 30 19:11:38.687111 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.687028 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr" event={"ID":"fa4187a2-d01f-487d-8ebe-5c835efc4eee","Type":"ContainerDied","Data":"74710e55d359fbed988c46c909ab3a101eaa4e703d2f054105ed9ec2f2956aa6"} Oct 30 19:11:38.687111 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.687052 4582 scope.go:122] "RemoveContainer" containerID="b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0" Oct 30 19:11:38.698138 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.698113 4582 scope.go:122] "RemoveContainer" containerID="b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0" Oct 30 19:11:38.698417 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:11:38.698400 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0\": container with ID starting with b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0 not found: ID does not exist" containerID="b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0" Oct 30 19:11:38.698493 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.698425 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0"} err="failed to get container status \"b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0\": rpc error: code = NotFound desc = could not find container \"b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0\": container with ID starting with b228da8f907d02efb834ec5d861900aea79191bfcd103d5ee1bb8f28f2a4cba0 not found: ID does not exist" Oct 30 19:11:38.739170 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.739142 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr"] Oct 30 19:11:38.742798 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:38.742776 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-hv9dr"] Oct 30 19:11:39.309656 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:39.309620 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" path="/var/lib/kubelet/pods/fa4187a2-d01f-487d-8ebe-5c835efc4eee/volumes" Oct 30 19:11:39.482283 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:39.482242 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:41.195331 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:41.195296 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:42.880330 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:42.880274 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:44.533973 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:44.533941 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:46.217260 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:46.217226 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:47.937885 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:47.937850 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:49.811345 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:49.811307 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:51.479167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:51.479125 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:53.199194 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:53.199157 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:54.943042 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:54.942989 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:56.770914 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:56.770879 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:58.405061 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:11:58.405023 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:00.156857 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:00.156790 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:01.958846 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:01.958795 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:03.646918 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:03.646887 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:05.474711 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:05.474673 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:06.225218 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:12:06.225176 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:12:07.266304 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:07.266270 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:09.000208 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:09.000175 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:10.666271 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:10.666227 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:12.529888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:12.529847 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:13.944524 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:13.944488 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:14.432378 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:14.432347 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:16.246761 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:16.246729 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:17.976125 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:17.976092 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:19.890243 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:19.890150 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:20.186925 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:20.186815 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:21.543649 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:21.543614 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:23.385470 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:23.385399 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:23.534682 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:23.534645 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv"] Oct 30 19:12:23.566679 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:23.566649 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:25.735642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:25.735606 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:26.686160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:26.686120 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:26.790151 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:26.790123 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:27.478508 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:27.478474 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:29.288690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:29.288662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:31.100747 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:31.100715 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:32.844602 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:32.844558 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:34.574966 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:34.574907 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:34.676745 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:34.676711 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:36.225294 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:12:36.225256 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:12:36.361748 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:36.361716 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:38.172837 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.172793 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:38.555378 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.555294 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" podUID="19653dd4-9014-4a62-a060-97a9dd1e6ba9" containerName="main" containerID="cri-o://88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc" gracePeriod=45 Oct 30 19:12:38.604169 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.604135 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" podUID="19653dd4-9014-4a62-a060-97a9dd1e6ba9" containerName="main" probeResult="failure" output="Get \"https://10.132.1.14:8000/ready\": dial tcp 10.132.1.14:8000: connect: connection refused" Oct 30 19:12:38.835775 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.835748 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:12:38.920518 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920486 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-home\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-home\") pod \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " Oct 30 19:12:38.920518 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920524 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tmp-dir\") pod \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " Oct 30 19:12:38.920752 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920559 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-model-cache\") pod \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " Oct 30 19:12:38.920752 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920612 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19653dd4-9014-4a62-a060-97a9dd1e6ba9-kube-api-access-pcvjq\" (UniqueName: \"kubernetes.io/projected/19653dd4-9014-4a62-a060-97a9dd1e6ba9-kube-api-access-pcvjq\") pod \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " Oct 30 19:12:38.920752 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920648 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-dshm\") pod \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " Oct 30 19:12:38.921011 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920779 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tmp-dir" pod "19653dd4-9014-4a62-a060-97a9dd1e6ba9" (UID: "19653dd4-9014-4a62-a060-97a9dd1e6ba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:38.921011 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920812 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-home" pod "19653dd4-9014-4a62-a060-97a9dd1e6ba9" (UID: "19653dd4-9014-4a62-a060-97a9dd1e6ba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:38.921011 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920871 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tls-certs\" (UniqueName: \"kubernetes.io/secret/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tls-certs\") pod \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\" (UID: \"19653dd4-9014-4a62-a060-97a9dd1e6ba9\") " Oct 30 19:12:38.921011 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.920896 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-model-cache" pod "19653dd4-9014-4a62-a060-97a9dd1e6ba9" (UID: "19653dd4-9014-4a62-a060-97a9dd1e6ba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:38.921281 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.921265 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.921327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.921306 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.921327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.921317 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.922975 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.922956 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tls-certs" pod "19653dd4-9014-4a62-a060-97a9dd1e6ba9" (UID: "19653dd4-9014-4a62-a060-97a9dd1e6ba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:38.923275 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.923247 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-dshm" pod "19653dd4-9014-4a62-a060-97a9dd1e6ba9" (UID: "19653dd4-9014-4a62-a060-97a9dd1e6ba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:38.923275 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:38.923258 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19653dd4-9014-4a62-a060-97a9dd1e6ba9-kube-api-access-pcvjq" pod "19653dd4-9014-4a62-a060-97a9dd1e6ba9" (UID: "19653dd4-9014-4a62-a060-97a9dd1e6ba9"). InnerVolumeSpecName "kube-api-access-pcvjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:39.022100 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.022072 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19653dd4-9014-4a62-a060-97a9dd1e6ba9-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:39.022100 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.022098 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvjq\" (UniqueName: \"kubernetes.io/projected/19653dd4-9014-4a62-a060-97a9dd1e6ba9-kube-api-access-pcvjq\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:39.022317 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.022107 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19653dd4-9014-4a62-a060-97a9dd1e6ba9-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:39.066430 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.066350 4582 generic.go:356] "Generic (PLEG): container finished" podID="19653dd4-9014-4a62-a060-97a9dd1e6ba9" containerID="88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc" exitCode=0 Oct 30 19:12:39.066430 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.066396 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" Oct 30 19:12:39.066603 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.066436 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" event={"ID":"19653dd4-9014-4a62-a060-97a9dd1e6ba9","Type":"ContainerDied","Data":"88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc"} Oct 30 19:12:39.066603 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.066480 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv" event={"ID":"19653dd4-9014-4a62-a060-97a9dd1e6ba9","Type":"ContainerDied","Data":"9e89c01e6fb95f26445a27ce4c4bee3b3f13e6800b05aa5fbdd0b55b4f57185c"} Oct 30 19:12:39.066603 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.066505 4582 scope.go:122] "RemoveContainer" containerID="88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc" Oct 30 19:12:39.077224 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.077209 4582 scope.go:122] "RemoveContainer" containerID="88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc" Oct 30 19:12:39.077478 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:12:39.077457 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc\": container with ID starting with 88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc not found: ID does not exist" containerID="88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc" Oct 30 19:12:39.077519 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.077490 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc"} err="failed to get container status \"88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc\": rpc error: code = NotFound desc = could not find container \"88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc\": container with ID starting with 88c401a022f6dbc25a0c657ea89347659b60e93983223532aa2d39356327e5cc not found: ID does not exist" Oct 30 19:12:39.121427 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.121402 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv"] Oct 30 19:12:39.124690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.124671 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-24b124/rate-test-model-24b124-kserve-5989f7f745-89gvv"] Oct 30 19:12:39.300516 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.300486 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19653dd4-9014-4a62-a060-97a9dd1e6ba9" path="/var/lib/kubelet/pods/19653dd4-9014-4a62-a060-97a9dd1e6ba9/volumes" Oct 30 19:12:39.936125 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:39.936092 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:41.663426 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:41.663392 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:43.330495 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:43.330461 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:45.408285 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:45.408250 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:45.798689 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:45.798603 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j"] Oct 30 19:12:45.798886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:45.798805 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" containerID="cri-o://aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff" gracePeriod=30 Oct 30 19:12:47.361891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:47.361857 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:49.086867 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:49.086833 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:50.837253 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:50.837216 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:51.073762 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.073703 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:12:51.144889 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.144856 4582 generic.go:356] "Generic (PLEG): container finished" podID="3d167647-e7ba-4ded-845e-e0737006aaab" containerID="aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff" exitCode=0 Oct 30 19:12:51.145062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.144899 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" event={"ID":"3d167647-e7ba-4ded-845e-e0737006aaab","Type":"ContainerDied","Data":"aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff"} Oct 30 19:12:51.145062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.144948 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" event={"ID":"3d167647-e7ba-4ded-845e-e0737006aaab","Type":"ContainerDied","Data":"c4a1cd7924745c4df0401f88a619b634afe6129dc50ff74bd274fa8423b7b0f5"} Oct 30 19:12:51.145062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.144959 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j" Oct 30 19:12:51.145062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.144979 4582 scope.go:122] "RemoveContainer" containerID="aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff" Oct 30 19:12:51.156797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.156775 4582 scope.go:122] "RemoveContainer" containerID="aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff" Oct 30 19:12:51.157134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157108 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-credential-socket\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.157232 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157204 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-kube-api-access-dgm5p\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-kube-api-access-dgm5p\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.157232 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:12:51.157108 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff\": container with ID starting with aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff not found: ID does not exist" containerID="aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff" Oct 30 19:12:51.157364 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157237 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d167647-e7ba-4ded-845e-e0737006aaab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d167647-e7ba-4ded-845e-e0737006aaab-istio-podinfo\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.157364 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157244 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff"} err="failed to get container status \"aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff\": rpc error: code = NotFound desc = could not find container \"aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff\": container with ID starting with aef9c4772a9e4fc93ac778342b0e41e6547ac01c719b2983128022672a4654ff not found: ID does not exist" Oct 30 19:12:51.157364 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157271 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d167647-e7ba-4ded-845e-e0737006aaab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d167647-e7ba-4ded-845e-e0737006aaab-istiod-ca-cert\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.157550 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157362 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-credential-socket" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.157789 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157763 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-istio-token\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-istio-token\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.157789 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157772 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d167647-e7ba-4ded-845e-e0737006aaab-istiod-ca-cert" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:51.157991 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157802 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-data\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.157991 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157882 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-certs\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.158117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.157994 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-socket\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.158117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158034 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-envoy\") pod \"3d167647-e7ba-4ded-845e-e0737006aaab\" (UID: \"3d167647-e7ba-4ded-845e-e0737006aaab\") " Oct 30 19:12:51.158246 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158123 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-data" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.158246 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158232 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-certs" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.158365 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158321 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-socket" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.158530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158508 4582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.158610 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158537 4582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-credential-socket\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.158610 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158553 4582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d167647-e7ba-4ded-845e-e0737006aaab-istiod-ca-cert\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.158610 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158567 4582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-data\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.158610 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.158580 4582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-workload-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.159678 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.159651 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-kube-api-access-dgm5p" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "kube-api-access-dgm5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.160002 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.159981 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d167647-e7ba-4ded-845e-e0737006aaab-istio-podinfo" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:12:51.160304 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.160280 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-istio-token" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:51.160545 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.160515 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-envoy" pod "3d167647-e7ba-4ded-845e-e0737006aaab" (UID: "3d167647-e7ba-4ded-845e-e0737006aaab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:51.260004 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.259969 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgm5p\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-kube-api-access-dgm5p\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.260004 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.260001 4582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d167647-e7ba-4ded-845e-e0737006aaab-istio-podinfo\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.260004 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.260010 4582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d167647-e7ba-4ded-845e-e0737006aaab-istio-token\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.260245 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.260019 4582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d167647-e7ba-4ded-845e-e0737006aaab-istio-envoy\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:12:51.286011 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.285976 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:51.373440 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.373411 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:51.484682 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.484651 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j"] Oct 30 19:12:51.496395 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:51.496366 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-76h9j"] Oct 30 19:12:52.531350 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:52.516740 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm"] Oct 30 19:12:52.540882 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:52.540850 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:52.659598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:52.659568 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:53.302565 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:53.302535 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" path="/var/lib/kubelet/pods/3d167647-e7ba-4ded-845e-e0737006aaab/volumes" Oct 30 19:12:54.365999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:54.365963 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:56.062326 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:56.062290 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:57.775586 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:57.775558 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:59.481491 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:12:59.481444 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:01.179311 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.179271 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg"] Oct 30 19:13:01.181803 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.179538 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" podUID="ce301cb9-9812-4f86-abd9-b1d41e8833ea" containerName="maas-api" containerID="cri-o://b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18" gracePeriod=30 Oct 30 19:13:01.239282 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.239251 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:01.443750 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.443693 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 19:13:01.572542 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.572510 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-metrics-certs\") pod \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " Oct 30 19:13:01.572726 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.572680 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce301cb9-9812-4f86-abd9-b1d41e8833ea-kube-api-access-sdlrw\" (UniqueName: \"kubernetes.io/projected/ce301cb9-9812-4f86-abd9-b1d41e8833ea-kube-api-access-sdlrw\") pod \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " Oct 30 19:13:01.572795 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.572760 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce301cb9-9812-4f86-abd9-b1d41e8833ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce301cb9-9812-4f86-abd9-b1d41e8833ea-openshift-service-ca\") pod \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " Oct 30 19:13:01.572898 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.572877 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-maas-api-tls\") pod \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\" (UID: \"ce301cb9-9812-4f86-abd9-b1d41e8833ea\") " Oct 30 19:13:01.573219 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.573188 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce301cb9-9812-4f86-abd9-b1d41e8833ea-openshift-service-ca" pod "ce301cb9-9812-4f86-abd9-b1d41e8833ea" (UID: "ce301cb9-9812-4f86-abd9-b1d41e8833ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:01.573360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.573329 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce301cb9-9812-4f86-abd9-b1d41e8833ea-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:01.574754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.574733 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-metrics-certs" pod "ce301cb9-9812-4f86-abd9-b1d41e8833ea" (UID: "ce301cb9-9812-4f86-abd9-b1d41e8833ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:01.574845 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.574763 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce301cb9-9812-4f86-abd9-b1d41e8833ea-kube-api-access-sdlrw" pod "ce301cb9-9812-4f86-abd9-b1d41e8833ea" (UID: "ce301cb9-9812-4f86-abd9-b1d41e8833ea"). InnerVolumeSpecName "kube-api-access-sdlrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:01.574905 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.574880 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-maas-api-tls" pod "ce301cb9-9812-4f86-abd9-b1d41e8833ea" (UID: "ce301cb9-9812-4f86-abd9-b1d41e8833ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:01.674021 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.673984 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:01.674021 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.674015 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce301cb9-9812-4f86-abd9-b1d41e8833ea-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:01.674021 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.674024 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdlrw\" (UniqueName: \"kubernetes.io/projected/ce301cb9-9812-4f86-abd9-b1d41e8833ea-kube-api-access-sdlrw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:01.922316 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:01.922283 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:02.040019 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.039981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:02.142549 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.142512 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:02.226754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.226670 4582 generic.go:356] "Generic (PLEG): container finished" podID="ce301cb9-9812-4f86-abd9-b1d41e8833ea" containerID="b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18" exitCode=0 Oct 30 19:13:02.226754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.226715 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" Oct 30 19:13:02.226754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.226736 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" event={"ID":"ce301cb9-9812-4f86-abd9-b1d41e8833ea","Type":"ContainerDied","Data":"b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18"} Oct 30 19:13:02.227378 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.226789 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg" event={"ID":"ce301cb9-9812-4f86-abd9-b1d41e8833ea","Type":"ContainerDied","Data":"068dcfb44a5c661f9fa99a249ed4cefd6ec45b911bb7dd56cfd68521d3f219fb"} Oct 30 19:13:02.227378 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.226834 4582 scope.go:122] "RemoveContainer" containerID="b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18" Oct 30 19:13:02.237712 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.237686 4582 scope.go:122] "RemoveContainer" containerID="b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18" Oct 30 19:13:02.237997 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:02.237979 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18\": container with ID starting with b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18 not found: ID does not exist" containerID="b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18" Oct 30 19:13:02.238072 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.238007 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18"} err="failed to get container status \"b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18\": rpc error: code = NotFound desc = could not find container \"b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18\": container with ID starting with b46ef3909a8f51a2f499d6e25c0fd4c3f2308c96a68e0166179ebf7384ab2e18 not found: ID does not exist" Oct 30 19:13:02.280100 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.280073 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg"] Oct 30 19:13:02.291777 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.291747 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1c4bcd-6b97fb4bb4-t8sbg"] Oct 30 19:13:02.724334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.724305 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:02.835536 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.835507 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l"] Oct 30 19:13:02.947598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:02.947564 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:03.205860 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:03.205816 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:03.301877 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:03.301849 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce301cb9-9812-4f86-abd9-b1d41e8833ea" path="/var/lib/kubelet/pods/ce301cb9-9812-4f86-abd9-b1d41e8833ea/volumes" Oct 30 19:13:03.676860 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:03.676798 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:04.324075 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:04.324034 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:04.487047 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:04.487015 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:06.225891 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:06.225850 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:13:06.767129 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:06.767089 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:07.544852 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.544796 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" podUID="7835c137-0861-48e2-bb3f-ea72ef3060da" containerName="main" containerID="cri-o://2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9" gracePeriod=45 Oct 30 19:13:07.816697 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.816672 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:13:07.946835 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.946795 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7835c137-0861-48e2-bb3f-ea72ef3060da-kube-api-access-flbp5\" (UniqueName: \"kubernetes.io/projected/7835c137-0861-48e2-bb3f-ea72ef3060da-kube-api-access-flbp5\") pod \"7835c137-0861-48e2-bb3f-ea72ef3060da\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " Oct 30 19:13:07.947007 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.946921 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-dshm\") pod \"7835c137-0861-48e2-bb3f-ea72ef3060da\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " Oct 30 19:13:07.947060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947016 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-model-cache\") pod \"7835c137-0861-48e2-bb3f-ea72ef3060da\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " Oct 30 19:13:07.947060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947049 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-home\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-home\") pod \"7835c137-0861-48e2-bb3f-ea72ef3060da\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " Oct 30 19:13:07.947192 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947167 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-tmp-dir\") pod \"7835c137-0861-48e2-bb3f-ea72ef3060da\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " Oct 30 19:13:07.947386 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947284 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7835c137-0861-48e2-bb3f-ea72ef3060da-tls-certs\" (UniqueName: \"kubernetes.io/secret/7835c137-0861-48e2-bb3f-ea72ef3060da-tls-certs\") pod \"7835c137-0861-48e2-bb3f-ea72ef3060da\" (UID: \"7835c137-0861-48e2-bb3f-ea72ef3060da\") " Oct 30 19:13:07.947386 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947311 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-model-cache" pod "7835c137-0861-48e2-bb3f-ea72ef3060da" (UID: "7835c137-0861-48e2-bb3f-ea72ef3060da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:07.947386 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947327 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-home" pod "7835c137-0861-48e2-bb3f-ea72ef3060da" (UID: "7835c137-0861-48e2-bb3f-ea72ef3060da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:07.947561 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947430 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-tmp-dir" pod "7835c137-0861-48e2-bb3f-ea72ef3060da" (UID: "7835c137-0861-48e2-bb3f-ea72ef3060da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:07.947758 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947742 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.947802 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947762 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.947802 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.947770 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.949219 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.949196 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7835c137-0861-48e2-bb3f-ea72ef3060da-tls-certs" pod "7835c137-0861-48e2-bb3f-ea72ef3060da" (UID: "7835c137-0861-48e2-bb3f-ea72ef3060da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:07.949368 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.949352 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7835c137-0861-48e2-bb3f-ea72ef3060da-kube-api-access-flbp5" pod "7835c137-0861-48e2-bb3f-ea72ef3060da" (UID: "7835c137-0861-48e2-bb3f-ea72ef3060da"). InnerVolumeSpecName "kube-api-access-flbp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:07.949431 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:07.949362 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-dshm" pod "7835c137-0861-48e2-bb3f-ea72ef3060da" (UID: "7835c137-0861-48e2-bb3f-ea72ef3060da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:08.048721 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.048642 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbp5\" (UniqueName: \"kubernetes.io/projected/7835c137-0861-48e2-bb3f-ea72ef3060da-kube-api-access-flbp5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:08.048721 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.048673 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7835c137-0861-48e2-bb3f-ea72ef3060da-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:08.048721 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.048682 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7835c137-0861-48e2-bb3f-ea72ef3060da-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:08.267336 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.267298 4582 generic.go:356] "Generic (PLEG): container finished" podID="7835c137-0861-48e2-bb3f-ea72ef3060da" containerID="2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9" exitCode=0 Oct 30 19:13:08.267526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.267352 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" Oct 30 19:13:08.267526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.267381 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" event={"ID":"7835c137-0861-48e2-bb3f-ea72ef3060da","Type":"ContainerDied","Data":"2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9"} Oct 30 19:13:08.267526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.267421 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm" event={"ID":"7835c137-0861-48e2-bb3f-ea72ef3060da","Type":"ContainerDied","Data":"f1d454e82bc4397d0f9d42b77ac248bf16454f44f91ab23f90aa7d5fc326ad3e"} Oct 30 19:13:08.267526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.267439 4582 scope.go:122] "RemoveContainer" containerID="2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9" Oct 30 19:13:08.277271 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.277256 4582 scope.go:122] "RemoveContainer" containerID="2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9" Oct 30 19:13:08.277531 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:08.277513 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9\": container with ID starting with 2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9 not found: ID does not exist" containerID="2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9" Oct 30 19:13:08.277607 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.277539 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9"} err="failed to get container status \"2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9\": rpc error: code = NotFound desc = could not find container \"2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9\": container with ID starting with 2c309e7fe616149a5140eb2ec545194c716c4b1c139f07c838db70f4ee9dedc9 not found: ID does not exist" Oct 30 19:13:08.318502 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.318427 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm"] Oct 30 19:13:08.323956 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.323928 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-675070/rate-test-model-675070-kserve-7559848f57-zfgnm"] Oct 30 19:13:08.482240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.482202 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:08.983285 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:08.983250 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:09.090115 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:09.090078 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:09.301240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:09.301168 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7835c137-0861-48e2-bb3f-ea72ef3060da" path="/var/lib/kubelet/pods/7835c137-0861-48e2-bb3f-ea72ef3060da/volumes" Oct 30 19:13:09.315577 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:09.315551 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:10.347957 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:10.347918 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:12.154415 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:12.154381 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:13.901184 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:13.901146 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:13.915979 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:13.915947 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn"] Oct 30 19:13:13.949916 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:13.949885 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg"] Oct 30 19:13:14.053666 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:14.053599 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:14.153158 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:14.153077 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:14.698972 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:14.698936 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t"] Oct 30 19:13:14.840157 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:14.840121 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:14.949273 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:14.949191 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:15.772979 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:15.772931 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:17.563001 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:17.562954 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:17.856577 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:17.856481 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" podUID="601ad845-dc26-4eb2-be78-4ef20d119b09" containerName="main" containerID="cri-o://703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7" gracePeriod=45 Oct 30 19:13:18.138212 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.138180 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:13:18.167248 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167217 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-model-cache\") pod \"601ad845-dc26-4eb2-be78-4ef20d119b09\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " Oct 30 19:13:18.167408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167297 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-home\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-home\") pod \"601ad845-dc26-4eb2-be78-4ef20d119b09\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " Oct 30 19:13:18.167408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167322 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/601ad845-dc26-4eb2-be78-4ef20d119b09-tls-certs\" (UniqueName: \"kubernetes.io/secret/601ad845-dc26-4eb2-be78-4ef20d119b09-tls-certs\") pod \"601ad845-dc26-4eb2-be78-4ef20d119b09\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " Oct 30 19:13:18.167408 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167403 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-tmp-dir\") pod \"601ad845-dc26-4eb2-be78-4ef20d119b09\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " Oct 30 19:13:18.167533 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167466 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601ad845-dc26-4eb2-be78-4ef20d119b09-kube-api-access-gvkxl\" (UniqueName: \"kubernetes.io/projected/601ad845-dc26-4eb2-be78-4ef20d119b09-kube-api-access-gvkxl\") pod \"601ad845-dc26-4eb2-be78-4ef20d119b09\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " Oct 30 19:13:18.167533 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167496 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-model-cache" pod "601ad845-dc26-4eb2-be78-4ef20d119b09" (UID: "601ad845-dc26-4eb2-be78-4ef20d119b09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.167533 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167508 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-dshm\") pod \"601ad845-dc26-4eb2-be78-4ef20d119b09\" (UID: \"601ad845-dc26-4eb2-be78-4ef20d119b09\") " Oct 30 19:13:18.167653 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167530 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-home" pod "601ad845-dc26-4eb2-be78-4ef20d119b09" (UID: "601ad845-dc26-4eb2-be78-4ef20d119b09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.167794 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.167772 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-tmp-dir" pod "601ad845-dc26-4eb2-be78-4ef20d119b09" (UID: "601ad845-dc26-4eb2-be78-4ef20d119b09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.168055 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.168037 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.168055 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.168055 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.168199 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.168066 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.169669 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.169644 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-dshm" pod "601ad845-dc26-4eb2-be78-4ef20d119b09" (UID: "601ad845-dc26-4eb2-be78-4ef20d119b09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:18.169783 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.169692 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601ad845-dc26-4eb2-be78-4ef20d119b09-tls-certs" pod "601ad845-dc26-4eb2-be78-4ef20d119b09" (UID: "601ad845-dc26-4eb2-be78-4ef20d119b09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:18.169783 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.169690 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601ad845-dc26-4eb2-be78-4ef20d119b09-kube-api-access-gvkxl" pod "601ad845-dc26-4eb2-be78-4ef20d119b09" (UID: "601ad845-dc26-4eb2-be78-4ef20d119b09"). InnerVolumeSpecName "kube-api-access-gvkxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:18.269046 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.269008 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkxl\" (UniqueName: \"kubernetes.io/projected/601ad845-dc26-4eb2-be78-4ef20d119b09-kube-api-access-gvkxl\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.269046 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.269039 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601ad845-dc26-4eb2-be78-4ef20d119b09-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.269046 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.269049 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601ad845-dc26-4eb2-be78-4ef20d119b09-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:18.332977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.332944 4582 generic.go:356] "Generic (PLEG): container finished" podID="601ad845-dc26-4eb2-be78-4ef20d119b09" containerID="703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7" exitCode=0 Oct 30 19:13:18.333152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.332987 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" Oct 30 19:13:18.333152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.332999 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" event={"ID":"601ad845-dc26-4eb2-be78-4ef20d119b09","Type":"ContainerDied","Data":"703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7"} Oct 30 19:13:18.333152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.333027 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l" event={"ID":"601ad845-dc26-4eb2-be78-4ef20d119b09","Type":"ContainerDied","Data":"7ef35200b8480963e1bfc62b4c13d2e26fba4263001a4b177d394b831ad6b907"} Oct 30 19:13:18.333152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.333045 4582 scope.go:122] "RemoveContainer" containerID="703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7" Oct 30 19:13:18.342814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.342789 4582 scope.go:122] "RemoveContainer" containerID="703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7" Oct 30 19:13:18.343095 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:18.343074 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7\": container with ID starting with 703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7 not found: ID does not exist" containerID="703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7" Oct 30 19:13:18.343165 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.343106 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7"} err="failed to get container status \"703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7\": rpc error: code = NotFound desc = could not find container \"703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7\": container with ID starting with 703ad53a2f9eeb1b7cd461d214014acfe085292532fa747f8080d16000e1e3d7 not found: ID does not exist" Oct 30 19:13:18.384288 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.384258 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l"] Oct 30 19:13:18.394071 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.394046 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-236fc4/sub-test-model-236fc4-kserve-58488d9669-jj98l"] Oct 30 19:13:18.916743 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.916702 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" podUID="fa546d21-3cfb-473f-a3fd-470fa68ad607" containerName="payload-processing" containerID="cri-o://c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b" gracePeriod=25 Oct 30 19:13:18.950297 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:18.950257 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" podUID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" containerName="payload-pre-processing" containerID="cri-o://27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242" gracePeriod=25 Oct 30 19:13:19.004866 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.004837 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77"] Oct 30 19:13:19.005065 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.005045 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" podUID="0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" containerName="maas-api" containerID="cri-o://6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c" gracePeriod=30 Oct 30 19:13:19.066457 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.066410 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:19.261012 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.260987 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 19:13:19.283235 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.283211 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-kube-api-access-b97r9\" (UniqueName: \"kubernetes.io/projected/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-kube-api-access-b97r9\") pod \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " Oct 30 19:13:19.283366 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.283259 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-maas-api-tls\") pod \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " Oct 30 19:13:19.283366 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.283360 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-metrics-certs\") pod \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " Oct 30 19:13:19.283452 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.283438 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-openshift-service-ca\") pod \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\" (UID: \"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89\") " Oct 30 19:13:19.283792 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.283765 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-openshift-service-ca" pod "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" (UID: "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:19.283933 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.283913 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.285379 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.285355 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-maas-api-tls" pod "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" (UID: "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:19.285474 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.285358 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-kube-api-access-b97r9" pod "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" (UID: "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89"). InnerVolumeSpecName "kube-api-access-b97r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:19.285520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.285477 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-metrics-certs" pod "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" (UID: "0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:19.305173 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.305141 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601ad845-dc26-4eb2-be78-4ef20d119b09" path="/var/lib/kubelet/pods/601ad845-dc26-4eb2-be78-4ef20d119b09/volumes" Oct 30 19:13:19.343783 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.343751 4582 generic.go:356] "Generic (PLEG): container finished" podID="0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" containerID="6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c" exitCode=0 Oct 30 19:13:19.343989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.343797 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" Oct 30 19:13:19.343989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.343851 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" event={"ID":"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89","Type":"ContainerDied","Data":"6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c"} Oct 30 19:13:19.343989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.343904 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77" event={"ID":"0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89","Type":"ContainerDied","Data":"7db85849e0e092574b58f89ae57095160b74d56f5a125238234ba41322aa243c"} Oct 30 19:13:19.343989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.343927 4582 scope.go:122] "RemoveContainer" containerID="6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c" Oct 30 19:13:19.353629 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.353604 4582 scope.go:122] "RemoveContainer" containerID="6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c" Oct 30 19:13:19.353915 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:19.353891 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c\": container with ID starting with 6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c not found: ID does not exist" containerID="6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c" Oct 30 19:13:19.353992 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.353927 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c"} err="failed to get container status \"6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c\": rpc error: code = NotFound desc = could not find container \"6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c\": container with ID starting with 6465c3ecb9bc8dba6c10150556d0109c9704a2cf611235f4e2d5d6f2d4ca921c not found: ID does not exist" Oct 30 19:13:19.385327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.385302 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.385327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.385326 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97r9\" (UniqueName: \"kubernetes.io/projected/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-kube-api-access-b97r9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.385487 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.385342 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.393338 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.393314 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77"] Oct 30 19:13:19.396280 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.396243 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d1a330-84d49dbb5c-7pj77"] Oct 30 19:13:19.699629 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.699587 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" podUID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" containerName="payload-processing" containerID="cri-o://651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b" gracePeriod=25 Oct 30 19:13:19.701977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.701951 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:19.801855 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.801802 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:19.915042 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:19.915009 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:20.313399 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:20.313364 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:20.414846 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:20.414807 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:20.560720 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:20.560686 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:20.664598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:20.664570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:21.300445 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:21.300411 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" path="/var/lib/kubelet/pods/0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89/volumes" Oct 30 19:13:21.882461 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:21.882422 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:23.616801 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:23.616769 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:24.073796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:24.073758 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" podUID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:24.074119 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:24.074089 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" podUID="fa546d21-3cfb-473f-a3fd-470fa68ad607" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:25.396886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:25.396854 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:26.796787 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:26.796753 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz"] Oct 30 19:13:26.797486 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:26.797453 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" podUID="eb8907ec-250b-4c8e-aae5-2274127baa43" containerName="maas-api" containerID="cri-o://7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a" gracePeriod=30 Oct 30 19:13:26.842558 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:26.842530 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:26.943031 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:26.942998 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:27.063601 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.063533 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 19:13:27.167989 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.167954 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-maas-api-tls\") pod \"eb8907ec-250b-4c8e-aae5-2274127baa43\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " Oct 30 19:13:27.168155 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.167996 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-metrics-certs\") pod \"eb8907ec-250b-4c8e-aae5-2274127baa43\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " Oct 30 19:13:27.168155 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.168067 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8907ec-250b-4c8e-aae5-2274127baa43-kube-api-access-wmkfn\" (UniqueName: \"kubernetes.io/projected/eb8907ec-250b-4c8e-aae5-2274127baa43-kube-api-access-wmkfn\") pod \"eb8907ec-250b-4c8e-aae5-2274127baa43\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " Oct 30 19:13:27.168234 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.168188 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8907ec-250b-4c8e-aae5-2274127baa43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8907ec-250b-4c8e-aae5-2274127baa43-openshift-service-ca\") pod \"eb8907ec-250b-4c8e-aae5-2274127baa43\" (UID: \"eb8907ec-250b-4c8e-aae5-2274127baa43\") " Oct 30 19:13:27.168599 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.168570 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8907ec-250b-4c8e-aae5-2274127baa43-openshift-service-ca" pod "eb8907ec-250b-4c8e-aae5-2274127baa43" (UID: "eb8907ec-250b-4c8e-aae5-2274127baa43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:27.170014 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.169985 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-maas-api-tls" pod "eb8907ec-250b-4c8e-aae5-2274127baa43" (UID: "eb8907ec-250b-4c8e-aae5-2274127baa43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:27.170184 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.170168 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8907ec-250b-4c8e-aae5-2274127baa43-kube-api-access-wmkfn" pod "eb8907ec-250b-4c8e-aae5-2274127baa43" (UID: "eb8907ec-250b-4c8e-aae5-2274127baa43"). InnerVolumeSpecName "kube-api-access-wmkfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:27.170270 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.170250 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-metrics-certs" pod "eb8907ec-250b-4c8e-aae5-2274127baa43" (UID: "eb8907ec-250b-4c8e-aae5-2274127baa43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:27.172879 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.172855 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:27.269456 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.269424 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:27.269456 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.269450 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8907ec-250b-4c8e-aae5-2274127baa43-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:27.269456 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.269460 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmkfn\" (UniqueName: \"kubernetes.io/projected/eb8907ec-250b-4c8e-aae5-2274127baa43-kube-api-access-wmkfn\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:27.269683 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.269469 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8907ec-250b-4c8e-aae5-2274127baa43-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:27.392685 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.392646 4582 generic.go:356] "Generic (PLEG): container finished" podID="eb8907ec-250b-4c8e-aae5-2274127baa43" containerID="7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a" exitCode=0 Oct 30 19:13:27.392910 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.392694 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" Oct 30 19:13:27.392910 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.392737 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" event={"ID":"eb8907ec-250b-4c8e-aae5-2274127baa43","Type":"ContainerDied","Data":"7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a"} Oct 30 19:13:27.392910 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.392796 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz" event={"ID":"eb8907ec-250b-4c8e-aae5-2274127baa43","Type":"ContainerDied","Data":"4745e458f026686820977867f73d36199abd56dde3efbed3483212f5204ab2fa"} Oct 30 19:13:27.392910 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.392852 4582 scope.go:122] "RemoveContainer" containerID="7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a" Oct 30 19:13:27.402949 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.402932 4582 scope.go:122] "RemoveContainer" containerID="7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a" Oct 30 19:13:27.403202 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:27.403185 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a\": container with ID starting with 7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a not found: ID does not exist" containerID="7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a" Oct 30 19:13:27.403272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.403209 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a"} err="failed to get container status \"7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a\": rpc error: code = NotFound desc = could not find container \"7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a\": container with ID starting with 7e3b6d3bfadc7d7dc813f019b45165d791e0e52c0a92cadc102f36dcfd5fb03a not found: ID does not exist" Oct 30 19:13:27.429599 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.429569 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" podUID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:27.444162 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.444141 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz"] Oct 30 19:13:27.447917 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.447897 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-713dc8-8bf9dfc65-xh4gz"] Oct 30 19:13:27.448710 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.448688 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:27.550650 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.550616 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:27.651718 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:27.651637 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:28.082336 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:28.082257 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:29.301066 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:29.301034 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8907ec-250b-4c8e-aae5-2274127baa43" path="/var/lib/kubelet/pods/eb8907ec-250b-4c8e-aae5-2274127baa43/volumes" Oct 30 19:13:29.494799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:29.494758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:31.274862 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:31.274802 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:32.972243 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:32.972211 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:34.074498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:34.074461 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" podUID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:34.075004 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:34.074521 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" podUID="fa546d21-3cfb-473f-a3fd-470fa68ad607" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:34.648394 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:34.648356 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:34.887631 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:34.887591 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:35.076326 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:35.076244 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:36.225011 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:36.224976 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:13:36.723119 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:36.723087 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:37.311596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:37.311564 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:37.415545 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:37.415511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:37.429457 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:37.429428 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" podUID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:38.396738 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:38.396667 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:38.494164 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:38.494113 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:39.075850 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.075793 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:39.176053 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.175904 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:39.225314 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.225291 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 19:13:39.234424 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.234398 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 19:13:39.269458 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.269427 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:39.297918 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.297889 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa546d21-3cfb-473f-a3fd-470fa68ad607-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa546d21-3cfb-473f-a3fd-470fa68ad607-plugins-config-volume\") pod \"fa546d21-3cfb-473f-a3fd-470fa68ad607\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " Oct 30 19:13:39.298104 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.297984 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa546d21-3cfb-473f-a3fd-470fa68ad607-kube-api-access-2xxxn\" (UniqueName: \"kubernetes.io/projected/fa546d21-3cfb-473f-a3fd-470fa68ad607-kube-api-access-2xxxn\") pod \"fa546d21-3cfb-473f-a3fd-470fa68ad607\" (UID: \"fa546d21-3cfb-473f-a3fd-470fa68ad607\") " Oct 30 19:13:39.298334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.298311 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa546d21-3cfb-473f-a3fd-470fa68ad607-plugins-config-volume" pod "fa546d21-3cfb-473f-a3fd-470fa68ad607" (UID: "fa546d21-3cfb-473f-a3fd-470fa68ad607"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:39.298456 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.298437 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa546d21-3cfb-473f-a3fd-470fa68ad607-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:39.300249 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.300225 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa546d21-3cfb-473f-a3fd-470fa68ad607-kube-api-access-2xxxn" pod "fa546d21-3cfb-473f-a3fd-470fa68ad607" (UID: "fa546d21-3cfb-473f-a3fd-470fa68ad607"). InnerVolumeSpecName "kube-api-access-2xxxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:39.398906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.398875 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-kube-api-access-x7jk4\" (UniqueName: \"kubernetes.io/projected/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-kube-api-access-x7jk4\") pod \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " Oct 30 19:13:39.399328 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.398931 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-plugins-config-volume\") pod \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\" (UID: \"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf\") " Oct 30 19:13:39.399407 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.399358 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-plugins-config-volume" pod "2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" (UID: "2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:39.399543 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.399522 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:39.399624 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.399549 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxxn\" (UniqueName: \"kubernetes.io/projected/fa546d21-3cfb-473f-a3fd-470fa68ad607-kube-api-access-2xxxn\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:39.401272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.401249 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-kube-api-access-x7jk4" pod "2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" (UID: "2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf"). InnerVolumeSpecName "kube-api-access-x7jk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:39.464524 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.464456 4582 generic.go:356] "Generic (PLEG): container finished" podID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" containerID="27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242" exitCode=0 Oct 30 19:13:39.464642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.464518 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" Oct 30 19:13:39.464642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.464536 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" event={"ID":"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf","Type":"ContainerDied","Data":"27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242"} Oct 30 19:13:39.464642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.464589 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg" event={"ID":"2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf","Type":"ContainerDied","Data":"0d31b4be821c2b5e731862b13de8c558dddf96f8f7b388539216984bbbe18411"} Oct 30 19:13:39.464642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.464622 4582 scope.go:122] "RemoveContainer" containerID="27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242" Oct 30 19:13:39.466047 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.466018 4582 generic.go:356] "Generic (PLEG): container finished" podID="fa546d21-3cfb-473f-a3fd-470fa68ad607" containerID="c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b" exitCode=0 Oct 30 19:13:39.466156 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.466061 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" Oct 30 19:13:39.466156 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.466103 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" event={"ID":"fa546d21-3cfb-473f-a3fd-470fa68ad607","Type":"ContainerDied","Data":"c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b"} Oct 30 19:13:39.466257 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.466155 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn" event={"ID":"fa546d21-3cfb-473f-a3fd-470fa68ad607","Type":"ContainerDied","Data":"6dc05e360f6c96ccd592c6449ebf1c8238851cfad58af62e8dde3dc0f7bf2523"} Oct 30 19:13:39.474673 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.474659 4582 scope.go:122] "RemoveContainer" containerID="27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242" Oct 30 19:13:39.474919 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:39.474897 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242\": container with ID starting with 27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242 not found: ID does not exist" containerID="27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242" Oct 30 19:13:39.475016 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.474923 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242"} err="failed to get container status \"27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242\": rpc error: code = NotFound desc = could not find container \"27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242\": container with ID starting with 27d396dbbbc01ab201b6e3e4f1ac03cfab55c1224660b36eceffbd77b6b9f242 not found: ID does not exist" Oct 30 19:13:39.475016 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.474939 4582 scope.go:122] "RemoveContainer" containerID="c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b" Oct 30 19:13:39.483318 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.483300 4582 scope.go:122] "RemoveContainer" containerID="c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b" Oct 30 19:13:39.483558 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:39.483537 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b\": container with ID starting with c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b not found: ID does not exist" containerID="c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b" Oct 30 19:13:39.483598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.483564 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b"} err="failed to get container status \"c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b\": rpc error: code = NotFound desc = could not find container \"c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b\": container with ID starting with c6c676e31d54b26d24a4bb7c56224f6777ae2ad66b0699d9e4f14ead7dd7f79b not found: ID does not exist" Oct 30 19:13:39.501086 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.501064 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7jk4\" (UniqueName: \"kubernetes.io/projected/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf-kube-api-access-x7jk4\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:39.518473 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.518448 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn"] Oct 30 19:13:39.525709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.525687 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d1a330-86c85f76d-mhswn"] Oct 30 19:13:39.556590 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.556564 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg"] Oct 30 19:13:39.560589 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.560571 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d1a330-6645f788c9-88bvg"] Oct 30 19:13:39.697057 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.697023 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:39.965868 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:39.965845 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 19:13:40.108852 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.108753 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-plugins-config-volume\") pod \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " Oct 30 19:13:40.108996 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.108868 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-kube-api-access-d7wn4\" (UniqueName: \"kubernetes.io/projected/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-kube-api-access-d7wn4\") pod \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\" (UID: \"1ab77db3-946a-49b1-bfa1-444f9ecd00ab\") " Oct 30 19:13:40.109143 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.109122 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-plugins-config-volume" pod "1ab77db3-946a-49b1-bfa1-444f9ecd00ab" (UID: "1ab77db3-946a-49b1-bfa1-444f9ecd00ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:40.110766 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.110747 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-kube-api-access-d7wn4" pod "1ab77db3-946a-49b1-bfa1-444f9ecd00ab" (UID: "1ab77db3-946a-49b1-bfa1-444f9ecd00ab"). InnerVolumeSpecName "kube-api-access-d7wn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:40.210370 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.210332 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.210370 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.210364 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wn4\" (UniqueName: \"kubernetes.io/projected/1ab77db3-946a-49b1-bfa1-444f9ecd00ab-kube-api-access-d7wn4\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.473374 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.473341 4582 generic.go:356] "Generic (PLEG): container finished" podID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" containerID="651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b" exitCode=0 Oct 30 19:13:40.473799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.473385 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" Oct 30 19:13:40.473799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.473415 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" event={"ID":"1ab77db3-946a-49b1-bfa1-444f9ecd00ab","Type":"ContainerDied","Data":"651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b"} Oct 30 19:13:40.473799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.473464 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t" event={"ID":"1ab77db3-946a-49b1-bfa1-444f9ecd00ab","Type":"ContainerDied","Data":"b92e9c5dac2fd0dd1e6d4eb0ae7c095721b420dbd5a1ea398261bdcf4cb98658"} Oct 30 19:13:40.473799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.473488 4582 scope.go:122] "RemoveContainer" containerID="651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b" Oct 30 19:13:40.483737 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.483720 4582 scope.go:122] "RemoveContainer" containerID="651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b" Oct 30 19:13:40.484031 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:40.484012 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b\": container with ID starting with 651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b not found: ID does not exist" containerID="651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b" Oct 30 19:13:40.484083 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.484041 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b"} err="failed to get container status \"651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b\": rpc error: code = NotFound desc = could not find container \"651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b\": container with ID starting with 651943f438fd304a56e721d1ad50fd4b6b06a1714f351b05744614d34e917f6b not found: ID does not exist" Oct 30 19:13:40.524679 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.524657 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t"] Oct 30 19:13:40.536266 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:40.536243 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-713dc8-7d557467f6-5nq4t"] Oct 30 19:13:41.170904 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:41.170864 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:41.300616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:41.300586 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" path="/var/lib/kubelet/pods/1ab77db3-946a-49b1-bfa1-444f9ecd00ab/volumes" Oct 30 19:13:41.301032 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:41.301017 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" path="/var/lib/kubelet/pods/2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf/volumes" Oct 30 19:13:41.301373 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:41.301361 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa546d21-3cfb-473f-a3fd-470fa68ad607" path="/var/lib/kubelet/pods/fa546d21-3cfb-473f-a3fd-470fa68ad607/volumes" Oct 30 19:13:43.000353 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:43.000321 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:44.197244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:44.197206 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:44.317549 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:44.317509 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:44.693861 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:44.693809 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:44.786190 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:44.786158 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:46.142307 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:46.142271 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp"] Oct 30 19:13:46.246193 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:46.246146 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:46.350658 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:46.350619 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:46.462715 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:46.462633 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:46.766722 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:46.766631 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:47.101368 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:47.101286 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:47.451346 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:47.451309 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:48.378665 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:48.378633 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:50.087355 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:50.087321 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:51.143238 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.143178 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" podUID="3b46eba5-ffed-4e96-ad32-dec766983bd2" containerName="payload-pre-processing" containerID="cri-o://58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496" gracePeriod=25 Oct 30 19:13:51.403621 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.403564 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 19:13:51.426077 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.426051 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b46eba5-ffed-4e96-ad32-dec766983bd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b46eba5-ffed-4e96-ad32-dec766983bd2-plugins-config-volume\") pod \"3b46eba5-ffed-4e96-ad32-dec766983bd2\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " Oct 30 19:13:51.426228 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.426108 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b46eba5-ffed-4e96-ad32-dec766983bd2-kube-api-access-97vfx\" (UniqueName: \"kubernetes.io/projected/3b46eba5-ffed-4e96-ad32-dec766983bd2-kube-api-access-97vfx\") pod \"3b46eba5-ffed-4e96-ad32-dec766983bd2\" (UID: \"3b46eba5-ffed-4e96-ad32-dec766983bd2\") " Oct 30 19:13:51.426453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.426431 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b46eba5-ffed-4e96-ad32-dec766983bd2-plugins-config-volume" pod "3b46eba5-ffed-4e96-ad32-dec766983bd2" (UID: "3b46eba5-ffed-4e96-ad32-dec766983bd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:51.426576 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.426561 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b46eba5-ffed-4e96-ad32-dec766983bd2-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:51.428204 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.428182 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b46eba5-ffed-4e96-ad32-dec766983bd2-kube-api-access-97vfx" pod "3b46eba5-ffed-4e96-ad32-dec766983bd2" (UID: "3b46eba5-ffed-4e96-ad32-dec766983bd2"). InnerVolumeSpecName "kube-api-access-97vfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:51.527201 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.527169 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97vfx\" (UniqueName: \"kubernetes.io/projected/3b46eba5-ffed-4e96-ad32-dec766983bd2-kube-api-access-97vfx\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:51.539514 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.539484 4582 generic.go:356] "Generic (PLEG): container finished" podID="3b46eba5-ffed-4e96-ad32-dec766983bd2" containerID="58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496" exitCode=0 Oct 30 19:13:51.539688 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.539531 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" Oct 30 19:13:51.539688 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.539556 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" event={"ID":"3b46eba5-ffed-4e96-ad32-dec766983bd2","Type":"ContainerDied","Data":"58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496"} Oct 30 19:13:51.539688 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.539604 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp" event={"ID":"3b46eba5-ffed-4e96-ad32-dec766983bd2","Type":"ContainerDied","Data":"b08a397e262abc6d588fa7cb82a4114c1779c1dc4d7f0a64fd70844600a49818"} Oct 30 19:13:51.539688 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.539626 4582 scope.go:122] "RemoveContainer" containerID="58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496" Oct 30 19:13:51.549361 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.549337 4582 scope.go:122] "RemoveContainer" containerID="58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496" Oct 30 19:13:51.549623 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:51.549603 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496\": container with ID starting with 58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496 not found: ID does not exist" containerID="58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496" Oct 30 19:13:51.549685 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.549631 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496"} err="failed to get container status \"58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496\": rpc error: code = NotFound desc = could not find container \"58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496\": container with ID starting with 58b5c7e44e620f2335e216318f96f7db7774985337393f7990ded36d9508a496 not found: ID does not exist" Oct 30 19:13:51.573347 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.573320 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd"] Oct 30 19:13:51.573517 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.573497 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" podUID="8956a23d-6d46-4674-8fc2-6d339cc20c6d" containerName="maas-api" containerID="cri-o://a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687" gracePeriod=30 Oct 30 19:13:51.599932 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.599907 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp"] Oct 30 19:13:51.612183 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.612156 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-39c99d-b5cd68c79-ltfgp"] Oct 30 19:13:51.656235 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.656146 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:51.813886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.813860 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 19:13:51.829499 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.829473 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-metrics-certs\") pod \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " Oct 30 19:13:51.829622 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.829524 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8956a23d-6d46-4674-8fc2-6d339cc20c6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8956a23d-6d46-4674-8fc2-6d339cc20c6d-openshift-service-ca\") pod \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " Oct 30 19:13:51.829688 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.829641 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8956a23d-6d46-4674-8fc2-6d339cc20c6d-kube-api-access-7s2l9\" (UniqueName: \"kubernetes.io/projected/8956a23d-6d46-4674-8fc2-6d339cc20c6d-kube-api-access-7s2l9\") pod \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " Oct 30 19:13:51.829786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.829772 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls\") pod \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\" (UID: \"8956a23d-6d46-4674-8fc2-6d339cc20c6d\") " Oct 30 19:13:51.829912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.829892 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8956a23d-6d46-4674-8fc2-6d339cc20c6d-openshift-service-ca" pod "8956a23d-6d46-4674-8fc2-6d339cc20c6d" (UID: "8956a23d-6d46-4674-8fc2-6d339cc20c6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:51.830199 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.830179 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8956a23d-6d46-4674-8fc2-6d339cc20c6d-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:51.831799 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.831772 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8956a23d-6d46-4674-8fc2-6d339cc20c6d-kube-api-access-7s2l9" pod "8956a23d-6d46-4674-8fc2-6d339cc20c6d" (UID: "8956a23d-6d46-4674-8fc2-6d339cc20c6d"). InnerVolumeSpecName "kube-api-access-7s2l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:51.831922 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.831869 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls" pod "8956a23d-6d46-4674-8fc2-6d339cc20c6d" (UID: "8956a23d-6d46-4674-8fc2-6d339cc20c6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:51.832156 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.832137 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-metrics-certs" pod "8956a23d-6d46-4674-8fc2-6d339cc20c6d" (UID: "8956a23d-6d46-4674-8fc2-6d339cc20c6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:51.931596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.931518 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:51.931596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.931545 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2l9\" (UniqueName: \"kubernetes.io/projected/8956a23d-6d46-4674-8fc2-6d339cc20c6d-kube-api-access-7s2l9\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:51.931596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:51.931556 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8956a23d-6d46-4674-8fc2-6d339cc20c6d-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.007911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.007872 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf"] Oct 30 19:13:52.008160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.008137 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" podUID="4053cb6c-e305-435c-bd77-8575aa05d772" containerName="maas-api" containerID="cri-o://82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0" gracePeriod=30 Oct 30 19:13:52.108869 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.108790 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:52.239080 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.239056 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:52.260400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.260376 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 19:13:52.342456 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.342423 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-metrics-certs\") pod \"4053cb6c-e305-435c-bd77-8575aa05d772\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " Oct 30 19:13:52.342634 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.342463 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls\") pod \"4053cb6c-e305-435c-bd77-8575aa05d772\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " Oct 30 19:13:52.342634 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.342613 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4053cb6c-e305-435c-bd77-8575aa05d772-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4053cb6c-e305-435c-bd77-8575aa05d772-openshift-service-ca\") pod \"4053cb6c-e305-435c-bd77-8575aa05d772\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " Oct 30 19:13:52.342757 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.342655 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4053cb6c-e305-435c-bd77-8575aa05d772-kube-api-access-vjbfl\" (UniqueName: \"kubernetes.io/projected/4053cb6c-e305-435c-bd77-8575aa05d772-kube-api-access-vjbfl\") pod \"4053cb6c-e305-435c-bd77-8575aa05d772\" (UID: \"4053cb6c-e305-435c-bd77-8575aa05d772\") " Oct 30 19:13:52.343051 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.343011 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4053cb6c-e305-435c-bd77-8575aa05d772-openshift-service-ca" pod "4053cb6c-e305-435c-bd77-8575aa05d772" (UID: "4053cb6c-e305-435c-bd77-8575aa05d772"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:52.344598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.344572 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls" pod "4053cb6c-e305-435c-bd77-8575aa05d772" (UID: "4053cb6c-e305-435c-bd77-8575aa05d772"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:52.344718 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.344597 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-metrics-certs" pod "4053cb6c-e305-435c-bd77-8575aa05d772" (UID: "4053cb6c-e305-435c-bd77-8575aa05d772"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:52.344765 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.344711 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4053cb6c-e305-435c-bd77-8575aa05d772-kube-api-access-vjbfl" pod "4053cb6c-e305-435c-bd77-8575aa05d772" (UID: "4053cb6c-e305-435c-bd77-8575aa05d772"). InnerVolumeSpecName "kube-api-access-vjbfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:52.387240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.387216 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:52.444257 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.444233 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.444257 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.444259 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4053cb6c-e305-435c-bd77-8575aa05d772-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.444400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.444272 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4053cb6c-e305-435c-bd77-8575aa05d772-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.444400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.444286 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbfl\" (UniqueName: \"kubernetes.io/projected/4053cb6c-e305-435c-bd77-8575aa05d772-kube-api-access-vjbfl\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.546919 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.546832 4582 generic.go:356] "Generic (PLEG): container finished" podID="4053cb6c-e305-435c-bd77-8575aa05d772" containerID="82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0" exitCode=0 Oct 30 19:13:52.546919 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.546881 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" Oct 30 19:13:52.546919 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.546907 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" event={"ID":"4053cb6c-e305-435c-bd77-8575aa05d772","Type":"ContainerDied","Data":"82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0"} Oct 30 19:13:52.547185 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.546955 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf" event={"ID":"4053cb6c-e305-435c-bd77-8575aa05d772","Type":"ContainerDied","Data":"37399f722e4d4d5e88e4a0669525785303f91e8eaeecdb16c1250b95aa516cc1"} Oct 30 19:13:52.547185 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.546982 4582 scope.go:122] "RemoveContainer" containerID="82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0" Oct 30 19:13:52.548393 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.548369 4582 generic.go:356] "Generic (PLEG): container finished" podID="8956a23d-6d46-4674-8fc2-6d339cc20c6d" containerID="a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687" exitCode=0 Oct 30 19:13:52.548511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.548408 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" Oct 30 19:13:52.548511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.548438 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" event={"ID":"8956a23d-6d46-4674-8fc2-6d339cc20c6d","Type":"ContainerDied","Data":"a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687"} Oct 30 19:13:52.548511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.548480 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd" event={"ID":"8956a23d-6d46-4674-8fc2-6d339cc20c6d","Type":"ContainerDied","Data":"c2bd2e9c49645b05ad04e07617a92f9753d921362809b09c24eb4ae3ecf6a819"} Oct 30 19:13:52.557260 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.557237 4582 scope.go:122] "RemoveContainer" containerID="82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0" Oct 30 19:13:52.557523 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:52.557502 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0\": container with ID starting with 82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0 not found: ID does not exist" containerID="82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0" Oct 30 19:13:52.557609 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.557531 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0"} err="failed to get container status \"82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0\": rpc error: code = NotFound desc = could not find container \"82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0\": container with ID starting with 82521aa465b600130d691e4b6ffc35201ce831b8ee207ce25226bf2f9c1ca8c0 not found: ID does not exist" Oct 30 19:13:52.557609 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.557548 4582 scope.go:122] "RemoveContainer" containerID="a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687" Oct 30 19:13:52.565984 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.565966 4582 scope.go:122] "RemoveContainer" containerID="a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687" Oct 30 19:13:52.566250 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:13:52.566231 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687\": container with ID starting with a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687 not found: ID does not exist" containerID="a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687" Oct 30 19:13:52.566325 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.566257 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687"} err="failed to get container status \"a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687\": rpc error: code = NotFound desc = could not find container \"a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687\": container with ID starting with a21d5480302d2675ee0c44ac212a8163d5006d3c97685d372f85caea12352687 not found: ID does not exist" Oct 30 19:13:52.596478 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.596455 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf"] Oct 30 19:13:52.599145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.599126 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-39c99d-795f875cd7-jpbvf"] Oct 30 19:13:52.633313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.633283 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd"] Oct 30 19:13:52.636058 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.636036 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6babca-9f7b98869-wpvmd"] Oct 30 19:13:52.777757 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.777721 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:52.878374 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:52.878345 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:53.082105 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:53.082071 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:53.300104 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:53.300030 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b46eba5-ffed-4e96-ad32-dec766983bd2" path="/var/lib/kubelet/pods/3b46eba5-ffed-4e96-ad32-dec766983bd2/volumes" Oct 30 19:13:53.300491 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:53.300477 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4053cb6c-e305-435c-bd77-8575aa05d772" path="/var/lib/kubelet/pods/4053cb6c-e305-435c-bd77-8575aa05d772/volumes" Oct 30 19:13:53.300888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:53.300874 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8956a23d-6d46-4674-8fc2-6d339cc20c6d" path="/var/lib/kubelet/pods/8956a23d-6d46-4674-8fc2-6d339cc20c6d/volumes" Oct 30 19:13:53.411893 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:53.411782 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:54.862933 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:54.862905 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:56.530095 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:56.530048 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:58.356814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:58.356774 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:59.177584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:59.177547 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:59.277737 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:13:59.277707 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:00.337545 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:00.337511 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:01.949284 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:01.949253 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:02.826856 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:02.826807 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:03.767956 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:03.767923 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:05.560883 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:05.560846 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:06.225180 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:06.225141 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:14:07.255312 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:07.255275 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:08.253861 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:08.253812 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p"] Oct 30 19:14:08.379472 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:08.379434 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:08.477314 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:08.477284 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.079712 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:09.079674 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.196632 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:09.196598 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.298356 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:09.298328 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.518530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:09.518498 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.616488 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:09.616455 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:10.083583 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:10.083556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:11.061237 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:11.061200 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:11.338058 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:11.337981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:12.728367 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:12.728328 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:12.829909 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:12.829879 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:12.931815 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:12.931778 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:13.254372 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:13.254333 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" podUID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" containerName="payload-processing" containerID="cri-o://d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9" gracePeriod=25 Oct 30 19:14:14.564013 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:14.563973 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:15.986977 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:15.986940 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z"] Oct 30 19:14:15.987436 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:15.987190 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" podUID="94045cce-08d4-4f95-8af7-5dd271d8ef96" containerName="maas-api" containerID="cri-o://5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d" gracePeriod=30 Oct 30 19:14:16.052224 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.052192 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:16.262153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.262116 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 19:14:16.406565 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.406536 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94045cce-08d4-4f95-8af7-5dd271d8ef96-kube-api-access-ckgbt\" (UniqueName: \"kubernetes.io/projected/94045cce-08d4-4f95-8af7-5dd271d8ef96-kube-api-access-ckgbt\") pod \"94045cce-08d4-4f95-8af7-5dd271d8ef96\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " Oct 30 19:14:16.406737 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.406654 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-maas-api-tls\") pod \"94045cce-08d4-4f95-8af7-5dd271d8ef96\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " Oct 30 19:14:16.406737 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.406675 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-metrics-certs\") pod \"94045cce-08d4-4f95-8af7-5dd271d8ef96\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " Oct 30 19:14:16.406737 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.406694 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94045cce-08d4-4f95-8af7-5dd271d8ef96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94045cce-08d4-4f95-8af7-5dd271d8ef96-openshift-service-ca\") pod \"94045cce-08d4-4f95-8af7-5dd271d8ef96\" (UID: \"94045cce-08d4-4f95-8af7-5dd271d8ef96\") " Oct 30 19:14:16.407207 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.407175 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94045cce-08d4-4f95-8af7-5dd271d8ef96-openshift-service-ca" pod "94045cce-08d4-4f95-8af7-5dd271d8ef96" (UID: "94045cce-08d4-4f95-8af7-5dd271d8ef96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:16.408848 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.408786 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-maas-api-tls" pod "94045cce-08d4-4f95-8af7-5dd271d8ef96" (UID: "94045cce-08d4-4f95-8af7-5dd271d8ef96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:16.408969 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.408858 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-metrics-certs" pod "94045cce-08d4-4f95-8af7-5dd271d8ef96" (UID: "94045cce-08d4-4f95-8af7-5dd271d8ef96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:16.408969 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.408883 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94045cce-08d4-4f95-8af7-5dd271d8ef96-kube-api-access-ckgbt" pod "94045cce-08d4-4f95-8af7-5dd271d8ef96" (UID: "94045cce-08d4-4f95-8af7-5dd271d8ef96"). InnerVolumeSpecName "kube-api-access-ckgbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:16.508490 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.508397 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:16.508490 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.508432 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94045cce-08d4-4f95-8af7-5dd271d8ef96-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:16.508490 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.508445 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94045cce-08d4-4f95-8af7-5dd271d8ef96-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:16.508490 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.508457 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckgbt\" (UniqueName: \"kubernetes.io/projected/94045cce-08d4-4f95-8af7-5dd271d8ef96-kube-api-access-ckgbt\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:16.577424 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.577384 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:16.684119 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.684081 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:16.687387 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.687356 4582 generic.go:356] "Generic (PLEG): container finished" podID="94045cce-08d4-4f95-8af7-5dd271d8ef96" containerID="5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d" exitCode=0 Oct 30 19:14:16.687511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.687400 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" Oct 30 19:14:16.687511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.687418 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" event={"ID":"94045cce-08d4-4f95-8af7-5dd271d8ef96","Type":"ContainerDied","Data":"5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d"} Oct 30 19:14:16.687511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.687466 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z" event={"ID":"94045cce-08d4-4f95-8af7-5dd271d8ef96","Type":"ContainerDied","Data":"c5059543d8aa819ce9fd840bd68a8f728399a798c89af1ee641029803e55d0a7"} Oct 30 19:14:16.687511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.687488 4582 scope.go:122] "RemoveContainer" containerID="5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d" Oct 30 19:14:16.697791 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.697774 4582 scope.go:122] "RemoveContainer" containerID="5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d" Oct 30 19:14:16.698061 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:16.698043 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d\": container with ID starting with 5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d not found: ID does not exist" containerID="5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d" Oct 30 19:14:16.698120 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.698067 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d"} err="failed to get container status \"5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d\": rpc error: code = NotFound desc = could not find container \"5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d\": container with ID starting with 5a7aa3ddfb8af4b0051d4dcf151d68ec13baf6e0cf909ad44054f4254f93333d not found: ID does not exist" Oct 30 19:14:16.741649 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.741617 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z"] Oct 30 19:14:16.745310 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.745278 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1a2436-6cc6844fd6-nbt7z"] Oct 30 19:14:16.784554 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:16.784474 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:17.178242 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:17.178206 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:17.299362 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:17.299333 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94045cce-08d4-4f95-8af7-5dd271d8ef96" path="/var/lib/kubelet/pods/94045cce-08d4-4f95-8af7-5dd271d8ef96/volumes" Oct 30 19:14:18.508255 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:18.508218 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:20.179342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:20.179299 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:20.431167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:20.431082 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:20.558133 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:20.558101 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr"] Oct 30 19:14:20.590447 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:20.590418 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f"] Oct 30 19:14:20.700771 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:20.700695 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:20.800670 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:20.800637 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:21.622840 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:21.622758 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:21.728201 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:21.728167 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:22.248990 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:22.248953 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:23.213544 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:23.213504 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" podUID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:23.935917 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:23.935879 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:24.052481 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:24.052448 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:24.283625 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:24.283551 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:25.559147 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.559101 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" podUID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" containerName="payload-processing" containerID="cri-o://b691225f9ba9e06dfa530b6507f6d20de4362a9f8164fea3365ffcc442a6988c" gracePeriod=25 Oct 30 19:14:25.573086 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.573050 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl"] Oct 30 19:14:25.573328 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.573300 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" podUID="21a928e3-74fc-442e-a69d-794dceb6774c" containerName="maas-api" containerID="cri-o://bc81e7f2f8bfec4c968cc9a344a13fd851d609b4a67dcb2815eb65f513f25871" gracePeriod=30 Oct 30 19:14:25.591318 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.590808 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" podUID="b2c87fb8-c070-4a6d-9aad-29237f31941b" containerName="payload-pre-processing" containerID="cri-o://3a05d0575f10252c7d6c266f592b17a45bb462afeea4c94bd387b3f03aa4ecb0" gracePeriod=25 Oct 30 19:14:25.643484 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.643418 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:25.741931 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.741899 4582 generic.go:356] "Generic (PLEG): container finished" podID="21a928e3-74fc-442e-a69d-794dceb6774c" containerID="bc81e7f2f8bfec4c968cc9a344a13fd851d609b4a67dcb2815eb65f513f25871" exitCode=0 Oct 30 19:14:25.742096 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.741947 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" event={"ID":"21a928e3-74fc-442e-a69d-794dceb6774c","Type":"ContainerDied","Data":"bc81e7f2f8bfec4c968cc9a344a13fd851d609b4a67dcb2815eb65f513f25871"} Oct 30 19:14:25.843707 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.843676 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 19:14:25.910972 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.910941 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a928e3-74fc-442e-a69d-794dceb6774c-kube-api-access-tx8h4\" (UniqueName: \"kubernetes.io/projected/21a928e3-74fc-442e-a69d-794dceb6774c-kube-api-access-tx8h4\") pod \"21a928e3-74fc-442e-a69d-794dceb6774c\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " Oct 30 19:14:25.911160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.911080 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-metrics-certs\") pod \"21a928e3-74fc-442e-a69d-794dceb6774c\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " Oct 30 19:14:25.911160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.911103 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a928e3-74fc-442e-a69d-794dceb6774c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a928e3-74fc-442e-a69d-794dceb6774c-openshift-service-ca\") pod \"21a928e3-74fc-442e-a69d-794dceb6774c\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " Oct 30 19:14:25.911160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.911125 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-maas-api-tls\") pod \"21a928e3-74fc-442e-a69d-794dceb6774c\" (UID: \"21a928e3-74fc-442e-a69d-794dceb6774c\") " Oct 30 19:14:25.911532 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.911506 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a928e3-74fc-442e-a69d-794dceb6774c-openshift-service-ca" pod "21a928e3-74fc-442e-a69d-794dceb6774c" (UID: "21a928e3-74fc-442e-a69d-794dceb6774c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:25.911636 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.911612 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a928e3-74fc-442e-a69d-794dceb6774c-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:25.913259 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.913234 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a928e3-74fc-442e-a69d-794dceb6774c-kube-api-access-tx8h4" pod "21a928e3-74fc-442e-a69d-794dceb6774c" (UID: "21a928e3-74fc-442e-a69d-794dceb6774c"). InnerVolumeSpecName "kube-api-access-tx8h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:25.913259 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.913239 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-metrics-certs" pod "21a928e3-74fc-442e-a69d-794dceb6774c" (UID: "21a928e3-74fc-442e-a69d-794dceb6774c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:25.913616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:25.913599 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-maas-api-tls" pod "21a928e3-74fc-442e-a69d-794dceb6774c" (UID: "21a928e3-74fc-442e-a69d-794dceb6774c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:26.012984 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.012951 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx8h4\" (UniqueName: \"kubernetes.io/projected/21a928e3-74fc-442e-a69d-794dceb6774c-kube-api-access-tx8h4\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:26.012984 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.012980 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:26.012984 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.012991 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a928e3-74fc-442e-a69d-794dceb6774c-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:26.108118 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.108038 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" podUID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:26.108265 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.108039 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" podUID="b2c87fb8-c070-4a6d-9aad-29237f31941b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:26.218583 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.218550 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:26.319801 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.319768 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:26.442563 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.442525 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:26.749213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.749136 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" Oct 30 19:14:26.749213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.749173 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl" event={"ID":"21a928e3-74fc-442e-a69d-794dceb6774c","Type":"ContainerDied","Data":"2bfc57586a5197622b8ea696cf93074484467db6a95fddc3c3699e3e29927335"} Oct 30 19:14:26.749668 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.749238 4582 scope.go:122] "RemoveContainer" containerID="bc81e7f2f8bfec4c968cc9a344a13fd851d609b4a67dcb2815eb65f513f25871" Oct 30 19:14:26.801255 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.801229 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl"] Oct 30 19:14:26.813296 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.813269 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-675070-c4bb7dd88-24bfl"] Oct 30 19:14:26.872399 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:26.872363 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:27.300152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:27.300118 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a928e3-74fc-442e-a69d-794dceb6774c" path="/var/lib/kubelet/pods/21a928e3-74fc-442e-a69d-794dceb6774c/volumes" Oct 30 19:14:28.355654 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:28.355617 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:30.171062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:30.171023 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:31.681522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:31.681486 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:32.216214 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:32.216157 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:32.326453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:32.326420 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:32.429511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:32.429476 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:32.821696 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:32.821665 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:33.213647 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.213607 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" podUID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:33.512685 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.512662 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 19:14:33.584213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.584183 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:33.593348 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.593323 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-kube-api-access-82bwf\" (UniqueName: \"kubernetes.io/projected/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-kube-api-access-82bwf\") pod \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " Oct 30 19:14:33.593455 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.593444 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-plugins-config-volume\") pod \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\" (UID: \"e5936235-bcfc-4c16-949c-b7e5a1b6fb35\") " Oct 30 19:14:33.593785 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.593763 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-plugins-config-volume" pod "e5936235-bcfc-4c16-949c-b7e5a1b6fb35" (UID: "e5936235-bcfc-4c16-949c-b7e5a1b6fb35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:33.595351 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.595330 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-kube-api-access-82bwf" pod "e5936235-bcfc-4c16-949c-b7e5a1b6fb35" (UID: "e5936235-bcfc-4c16-949c-b7e5a1b6fb35"). InnerVolumeSpecName "kube-api-access-82bwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:33.695010 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.694974 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.695010 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.695004 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bwf\" (UniqueName: \"kubernetes.io/projected/e5936235-bcfc-4c16-949c-b7e5a1b6fb35-kube-api-access-82bwf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.704838 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.704794 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:33.793509 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.793413 4582 generic.go:356] "Generic (PLEG): container finished" podID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" containerID="d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9" exitCode=0 Oct 30 19:14:33.793796 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.793470 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" Oct 30 19:14:33.794677 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.793489 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" event={"ID":"e5936235-bcfc-4c16-949c-b7e5a1b6fb35","Type":"ContainerDied","Data":"d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9"} Oct 30 19:14:33.794677 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.793890 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p" event={"ID":"e5936235-bcfc-4c16-949c-b7e5a1b6fb35","Type":"ContainerDied","Data":"6c6493b006f77277bc4ccff7fcbf13b7bc690a0591e9f82432032aa48179dcfc"} Oct 30 19:14:33.794677 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.793937 4582 scope.go:122] "RemoveContainer" containerID="d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9" Oct 30 19:14:33.804767 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.804746 4582 scope.go:122] "RemoveContainer" containerID="d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9" Oct 30 19:14:33.805113 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:33.805088 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9\": container with ID starting with d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9 not found: ID does not exist" containerID="d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9" Oct 30 19:14:33.805201 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.805123 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9"} err="failed to get container status \"d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9\": rpc error: code = NotFound desc = could not find container \"d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9\": container with ID starting with d7ff381ec005f9c0433994a143740fad20195b5db97d4257068a7921770604f9 not found: ID does not exist" Oct 30 19:14:33.826286 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:33.826252 4582 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5936235_bcfc_4c16_949c_b7e5a1b6fb35.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:14:33.826751 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:33.826264 4582 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5936235_bcfc_4c16_949c_b7e5a1b6fb35.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:14:33.848263 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.848233 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p"] Oct 30 19:14:33.850217 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.850199 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1a2436-567f469f4-qmp5p"] Oct 30 19:14:33.931975 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:33.931944 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:34.487721 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:34.487687 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:35.299959 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:35.299928 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" path="/var/lib/kubelet/pods/e5936235-bcfc-4c16-949c-b7e5a1b6fb35/volumes" Oct 30 19:14:35.389565 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:35.389525 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c"] Oct 30 19:14:35.587441 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:35.587350 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:35.692468 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:35.692436 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:36.108191 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:36.108149 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" podUID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:36.108191 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:36.108173 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" podUID="b2c87fb8-c070-4a6d-9aad-29237f31941b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:36.224527 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:36.224488 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:14:36.477965 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:36.477932 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:38.301535 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:38.301504 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:39.406409 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:39.406371 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:39.535518 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:39.535482 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:39.749940 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:39.749867 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:40.165585 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:40.165552 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:40.390259 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:40.390189 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" podUID="66ff8650-d22d-4c6f-a351-fcb70e40980f" containerName="payload-processing" containerID="cri-o://f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff" gracePeriod=25 Oct 30 19:14:42.140611 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:42.140570 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:42.760986 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:42.760951 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" podUID="66ff8650-d22d-4c6f-a351-fcb70e40980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:44.199644 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:44.199600 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:45.111390 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.111355 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:45.223729 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.223698 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:45.717624 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.716747 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh"] Oct 30 19:14:45.717624 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.717008 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" podUID="cc76d911-27d1-4fa7-b986-47ccdb9b6c43" containerName="maas-api" containerID="cri-o://2489f10e6461cf2b76df1840713ac0ecd5f4c278ec0c5ee2ee2679baf05450a9" gracePeriod=30 Oct 30 19:14:45.771864 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.771809 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:45.879489 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.879457 4582 generic.go:356] "Generic (PLEG): container finished" podID="cc76d911-27d1-4fa7-b986-47ccdb9b6c43" containerID="2489f10e6461cf2b76df1840713ac0ecd5f4c278ec0c5ee2ee2679baf05450a9" exitCode=0 Oct 30 19:14:45.879657 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.879576 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" event={"ID":"cc76d911-27d1-4fa7-b986-47ccdb9b6c43","Type":"ContainerDied","Data":"2489f10e6461cf2b76df1840713ac0ecd5f4c278ec0c5ee2ee2679baf05450a9"} Oct 30 19:14:45.881642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.881471 4582 generic.go:356] "Generic (PLEG): container finished" podID="b2c87fb8-c070-4a6d-9aad-29237f31941b" containerID="3a05d0575f10252c7d6c266f592b17a45bb462afeea4c94bd387b3f03aa4ecb0" exitCode=0 Oct 30 19:14:45.881642 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.881580 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" event={"ID":"b2c87fb8-c070-4a6d-9aad-29237f31941b","Type":"ContainerDied","Data":"3a05d0575f10252c7d6c266f592b17a45bb462afeea4c94bd387b3f03aa4ecb0"} Oct 30 19:14:45.883349 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.883201 4582 generic.go:356] "Generic (PLEG): container finished" podID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" containerID="b691225f9ba9e06dfa530b6507f6d20de4362a9f8164fea3365ffcc442a6988c" exitCode=0 Oct 30 19:14:45.883349 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.883272 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" event={"ID":"824ccad0-a794-4ae6-ab9c-b5cf75fda61f","Type":"ContainerDied","Data":"b691225f9ba9e06dfa530b6507f6d20de4362a9f8164fea3365ffcc442a6988c"} Oct 30 19:14:45.883349 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.883308 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" event={"ID":"824ccad0-a794-4ae6-ab9c-b5cf75fda61f","Type":"ContainerDied","Data":"ad4a26913d3d5eb52b3f11eaedf7587b9874de3e7b50bab885867bcdca1e9161"} Oct 30 19:14:45.883349 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.883332 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4a26913d3d5eb52b3f11eaedf7587b9874de3e7b50bab885867bcdca1e9161" Oct 30 19:14:45.887244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.887226 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 19:14:45.896522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.896502 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 19:14:45.968458 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:45.967872 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 19:14:46.035759 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.035729 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-plugins-config-volume\") pod \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " Oct 30 19:14:46.035958 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.035856 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-kube-api-access-m2nqf\" (UniqueName: \"kubernetes.io/projected/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-kube-api-access-m2nqf\") pod \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\" (UID: \"824ccad0-a794-4ae6-ab9c-b5cf75fda61f\") " Oct 30 19:14:46.035958 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.035914 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c87fb8-c070-4a6d-9aad-29237f31941b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c87fb8-c070-4a6d-9aad-29237f31941b-plugins-config-volume\") pod \"b2c87fb8-c070-4a6d-9aad-29237f31941b\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " Oct 30 19:14:46.036179 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.036073 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c87fb8-c070-4a6d-9aad-29237f31941b-kube-api-access-d985j\" (UniqueName: \"kubernetes.io/projected/b2c87fb8-c070-4a6d-9aad-29237f31941b-kube-api-access-d985j\") pod \"b2c87fb8-c070-4a6d-9aad-29237f31941b\" (UID: \"b2c87fb8-c070-4a6d-9aad-29237f31941b\") " Oct 30 19:14:46.036342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.036311 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-plugins-config-volume" pod "824ccad0-a794-4ae6-ab9c-b5cf75fda61f" (UID: "824ccad0-a794-4ae6-ab9c-b5cf75fda61f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:46.036467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.036379 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c87fb8-c070-4a6d-9aad-29237f31941b-plugins-config-volume" pod "b2c87fb8-c070-4a6d-9aad-29237f31941b" (UID: "b2c87fb8-c070-4a6d-9aad-29237f31941b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:46.036564 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.036544 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.036616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.036569 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c87fb8-c070-4a6d-9aad-29237f31941b-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.038219 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.038197 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c87fb8-c070-4a6d-9aad-29237f31941b-kube-api-access-d985j" pod "b2c87fb8-c070-4a6d-9aad-29237f31941b" (UID: "b2c87fb8-c070-4a6d-9aad-29237f31941b"). InnerVolumeSpecName "kube-api-access-d985j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:46.038313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.038230 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-kube-api-access-m2nqf" pod "824ccad0-a794-4ae6-ab9c-b5cf75fda61f" (UID: "824ccad0-a794-4ae6-ab9c-b5cf75fda61f"). InnerVolumeSpecName "kube-api-access-m2nqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:46.137212 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137173 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-openshift-service-ca\") pod \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " Oct 30 19:14:46.137431 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137243 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-metrics-certs\") pod \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " Oct 30 19:14:46.137431 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137293 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-kube-api-access-kqmgp\" (UniqueName: \"kubernetes.io/projected/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-kube-api-access-kqmgp\") pod \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " Oct 30 19:14:46.137431 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137332 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-maas-api-tls\") pod \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\" (UID: \"cc76d911-27d1-4fa7-b986-47ccdb9b6c43\") " Oct 30 19:14:46.137621 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137543 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-openshift-service-ca" pod "cc76d911-27d1-4fa7-b986-47ccdb9b6c43" (UID: "cc76d911-27d1-4fa7-b986-47ccdb9b6c43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:46.137751 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137731 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d985j\" (UniqueName: \"kubernetes.io/projected/b2c87fb8-c070-4a6d-9aad-29237f31941b-kube-api-access-d985j\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.137814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137755 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.137814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.137768 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2nqf\" (UniqueName: \"kubernetes.io/projected/824ccad0-a794-4ae6-ab9c-b5cf75fda61f-kube-api-access-m2nqf\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.140124 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.140094 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-metrics-certs" pod "cc76d911-27d1-4fa7-b986-47ccdb9b6c43" (UID: "cc76d911-27d1-4fa7-b986-47ccdb9b6c43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:46.140288 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.140136 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-kube-api-access-kqmgp" pod "cc76d911-27d1-4fa7-b986-47ccdb9b6c43" (UID: "cc76d911-27d1-4fa7-b986-47ccdb9b6c43"). InnerVolumeSpecName "kube-api-access-kqmgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:46.140288 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.140144 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-maas-api-tls" pod "cc76d911-27d1-4fa7-b986-47ccdb9b6c43" (UID: "cc76d911-27d1-4fa7-b986-47ccdb9b6c43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:46.153206 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.153175 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq"] Oct 30 19:14:46.153410 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.153385 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" podUID="d57db115-c40a-49da-bc23-bb87da91713d" containerName="maas-api" containerID="cri-o://11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23" gracePeriod=30 Oct 30 19:14:46.238361 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.238326 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.238361 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.238359 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqmgp\" (UniqueName: \"kubernetes.io/projected/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-kube-api-access-kqmgp\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.238781 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.238372 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc76d911-27d1-4fa7-b986-47ccdb9b6c43-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.270742 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.270718 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:46.392917 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.392857 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 19:14:46.440048 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.440009 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57db115-c40a-49da-bc23-bb87da91713d-kube-api-access-n95mm\" (UniqueName: \"kubernetes.io/projected/d57db115-c40a-49da-bc23-bb87da91713d-kube-api-access-n95mm\") pod \"d57db115-c40a-49da-bc23-bb87da91713d\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " Oct 30 19:14:46.440220 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.440096 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-maas-api-tls\") pod \"d57db115-c40a-49da-bc23-bb87da91713d\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " Oct 30 19:14:46.440272 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.440220 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57db115-c40a-49da-bc23-bb87da91713d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d57db115-c40a-49da-bc23-bb87da91713d-openshift-service-ca\") pod \"d57db115-c40a-49da-bc23-bb87da91713d\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " Oct 30 19:14:46.440670 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.440637 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57db115-c40a-49da-bc23-bb87da91713d-openshift-service-ca" pod "d57db115-c40a-49da-bc23-bb87da91713d" (UID: "d57db115-c40a-49da-bc23-bb87da91713d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:46.442343 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.442312 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57db115-c40a-49da-bc23-bb87da91713d-kube-api-access-n95mm" pod "d57db115-c40a-49da-bc23-bb87da91713d" (UID: "d57db115-c40a-49da-bc23-bb87da91713d"). InnerVolumeSpecName "kube-api-access-n95mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:46.442468 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.442418 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-maas-api-tls" pod "d57db115-c40a-49da-bc23-bb87da91713d" (UID: "d57db115-c40a-49da-bc23-bb87da91713d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:46.540964 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.540930 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-metrics-certs\") pod \"d57db115-c40a-49da-bc23-bb87da91713d\" (UID: \"d57db115-c40a-49da-bc23-bb87da91713d\") " Oct 30 19:14:46.541213 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.541200 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95mm\" (UniqueName: \"kubernetes.io/projected/d57db115-c40a-49da-bc23-bb87da91713d-kube-api-access-n95mm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.541271 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.541215 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.541271 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.541225 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d57db115-c40a-49da-bc23-bb87da91713d-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.542961 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.542935 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-metrics-certs" pod "d57db115-c40a-49da-bc23-bb87da91713d" (UID: "d57db115-c40a-49da-bc23-bb87da91713d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:46.641913 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.641870 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d57db115-c40a-49da-bc23-bb87da91713d-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:46.816790 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.816704 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:46.890115 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.890080 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" event={"ID":"cc76d911-27d1-4fa7-b986-47ccdb9b6c43","Type":"ContainerDied","Data":"e28ad65d44f04727fdce5406eaee76ff58b35b84d32ff1bdcaf6d6cf0822fc7b"} Oct 30 19:14:46.890300 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.890143 4582 scope.go:122] "RemoveContainer" containerID="2489f10e6461cf2b76df1840713ac0ecd5f4c278ec0c5ee2ee2679baf05450a9" Oct 30 19:14:46.890300 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.890090 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh" Oct 30 19:14:46.891641 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.891401 4582 generic.go:356] "Generic (PLEG): container finished" podID="d57db115-c40a-49da-bc23-bb87da91713d" containerID="11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23" exitCode=0 Oct 30 19:14:46.891641 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.891459 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" Oct 30 19:14:46.891868 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.891808 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" event={"ID":"d57db115-c40a-49da-bc23-bb87da91713d","Type":"ContainerDied","Data":"11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23"} Oct 30 19:14:46.892038 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.891900 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq" event={"ID":"d57db115-c40a-49da-bc23-bb87da91713d","Type":"ContainerDied","Data":"bfed43214b83295432761c5bae3c309a670fa83f087949aa45dc9e69d22e13b8"} Oct 30 19:14:46.893158 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.893117 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" Oct 30 19:14:46.893283 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.893182 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f" event={"ID":"b2c87fb8-c070-4a6d-9aad-29237f31941b","Type":"ContainerDied","Data":"fd28c517bb5866ddd6b7cf17b25ea59f0f7f5ae3081e0eb8114e49f441749673"} Oct 30 19:14:46.893362 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.893348 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr" Oct 30 19:14:46.902939 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.902495 4582 scope.go:122] "RemoveContainer" containerID="11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23" Oct 30 19:14:46.913623 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.913608 4582 scope.go:122] "RemoveContainer" containerID="11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23" Oct 30 19:14:46.913831 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:46.913803 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23\": container with ID starting with 11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23 not found: ID does not exist" containerID="11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23" Oct 30 19:14:46.913906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.913848 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23"} err="failed to get container status \"11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23\": rpc error: code = NotFound desc = could not find container \"11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23\": container with ID starting with 11de429548a96466cb8f46cd7e9d52462404042233b286a37100a5f4ff45eb23 not found: ID does not exist" Oct 30 19:14:46.913906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.913875 4582 scope.go:122] "RemoveContainer" containerID="3a05d0575f10252c7d6c266f592b17a45bb462afeea4c94bd387b3f03aa4ecb0" Oct 30 19:14:46.979459 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.979425 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:46.989833 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:46.989795 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f"] Oct 30 19:14:47.001868 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.001839 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-675070-7d6bd9bf67-4xv4f"] Oct 30 19:14:47.032251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.032221 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh"] Oct 30 19:14:47.040979 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.040956 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-09ef1d-55c948cdfc-b6jfh"] Oct 30 19:14:47.088046 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.087954 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr"] Oct 30 19:14:47.095913 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.095886 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-675070-5bf74cf574-r4wfr"] Oct 30 19:14:47.112876 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.112815 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:47.129702 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.129676 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq"] Oct 30 19:14:47.138669 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.138644 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-24b124-75cff98c5f-7x8cq"] Oct 30 19:14:47.300062 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.300024 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" path="/var/lib/kubelet/pods/824ccad0-a794-4ae6-ab9c-b5cf75fda61f/volumes" Oct 30 19:14:47.300589 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.300567 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c87fb8-c070-4a6d-9aad-29237f31941b" path="/var/lib/kubelet/pods/b2c87fb8-c070-4a6d-9aad-29237f31941b/volumes" Oct 30 19:14:47.301044 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.301025 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc76d911-27d1-4fa7-b986-47ccdb9b6c43" path="/var/lib/kubelet/pods/cc76d911-27d1-4fa7-b986-47ccdb9b6c43/volumes" Oct 30 19:14:47.301743 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.301723 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57db115-c40a-49da-bc23-bb87da91713d" path="/var/lib/kubelet/pods/d57db115-c40a-49da-bc23-bb87da91713d/volumes" Oct 30 19:14:47.617239 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:47.617205 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:49.086637 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:49.086603 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:50.852421 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:50.852381 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:50.953476 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:50.953442 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:51.054079 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:51.054049 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:52.637086 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:52.637050 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:52.761595 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:52.761550 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" podUID="66ff8650-d22d-4c6f-a351-fcb70e40980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:53.847966 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:53.847933 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:53.955522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:53.955487 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:54.224395 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:54.224354 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:54.391854 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:54.391804 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:54.662604 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:54.662572 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:56.195898 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.195859 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z"] Oct 30 19:14:56.196438 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.196110 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" podUID="50e150fc-db48-4659-9c2f-4bf1f6d4318d" containerName="maas-api" containerID="cri-o://1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde" gracePeriod=30 Oct 30 19:14:56.246530 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.246493 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:56.456657 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.456595 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 19:14:56.482350 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.482320 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:56.537461 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.537423 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e150fc-db48-4659-9c2f-4bf1f6d4318d-kube-api-access-qdqgh\" (UniqueName: \"kubernetes.io/projected/50e150fc-db48-4659-9c2f-4bf1f6d4318d-kube-api-access-qdqgh\") pod \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " Oct 30 19:14:56.537663 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.537503 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e150fc-db48-4659-9c2f-4bf1f6d4318d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e150fc-db48-4659-9c2f-4bf1f6d4318d-openshift-service-ca\") pod \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " Oct 30 19:14:56.537663 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.537632 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-metrics-certs\") pod \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " Oct 30 19:14:56.537804 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.537676 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls\") pod \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\" (UID: \"50e150fc-db48-4659-9c2f-4bf1f6d4318d\") " Oct 30 19:14:56.537942 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.537907 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e150fc-db48-4659-9c2f-4bf1f6d4318d-openshift-service-ca" pod "50e150fc-db48-4659-9c2f-4bf1f6d4318d" (UID: "50e150fc-db48-4659-9c2f-4bf1f6d4318d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:56.538273 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.538254 4582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e150fc-db48-4659-9c2f-4bf1f6d4318d-openshift-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:56.539731 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.539705 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e150fc-db48-4659-9c2f-4bf1f6d4318d-kube-api-access-qdqgh" pod "50e150fc-db48-4659-9c2f-4bf1f6d4318d" (UID: "50e150fc-db48-4659-9c2f-4bf1f6d4318d"). InnerVolumeSpecName "kube-api-access-qdqgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:56.539912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.539892 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls" pod "50e150fc-db48-4659-9c2f-4bf1f6d4318d" (UID: "50e150fc-db48-4659-9c2f-4bf1f6d4318d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:56.540217 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.540201 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-metrics-certs" pod "50e150fc-db48-4659-9c2f-4bf1f6d4318d" (UID: "50e150fc-db48-4659-9c2f-4bf1f6d4318d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:56.593583 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.593546 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:56.639900 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.639855 4582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-metrics-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:56.639900 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.639887 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e150fc-db48-4659-9c2f-4bf1f6d4318d-maas-api-tls\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:56.639900 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.639900 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdqgh\" (UniqueName: \"kubernetes.io/projected/50e150fc-db48-4659-9c2f-4bf1f6d4318d-kube-api-access-qdqgh\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:14:56.948170 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.948130 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:56.953268 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.953242 4582 generic.go:356] "Generic (PLEG): container finished" podID="50e150fc-db48-4659-9c2f-4bf1f6d4318d" containerID="1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde" exitCode=0 Oct 30 19:14:56.953400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.953283 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" Oct 30 19:14:56.953400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.953327 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" event={"ID":"50e150fc-db48-4659-9c2f-4bf1f6d4318d","Type":"ContainerDied","Data":"1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde"} Oct 30 19:14:56.953400 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.953377 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z" event={"ID":"50e150fc-db48-4659-9c2f-4bf1f6d4318d","Type":"ContainerDied","Data":"86fcdea900e72faa5ee2493650828e92302fe0e9e9083ed2c8a8cc1888a8a591"} Oct 30 19:14:56.953526 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.953400 4582 scope.go:122] "RemoveContainer" containerID="1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde" Oct 30 19:14:56.963154 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.963139 4582 scope.go:122] "RemoveContainer" containerID="1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde" Oct 30 19:14:56.963404 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:14:56.963389 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde\": container with ID starting with 1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde not found: ID does not exist" containerID="1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde" Oct 30 19:14:56.963470 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:56.963411 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde"} err="failed to get container status \"1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde\": rpc error: code = NotFound desc = could not find container \"1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde\": container with ID starting with 1282bf319cc60dc8ce7ec86ed15915d402fa81863dc9791a2053afaa9573bdde not found: ID does not exist" Oct 30 19:14:57.006174 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:57.006152 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z"] Oct 30 19:14:57.018023 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:57.018001 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c095fa-67f8f749db-r756z"] Oct 30 19:14:57.048264 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:57.048239 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:57.150583 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:57.150556 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:57.299113 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:57.299037 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e150fc-db48-4659-9c2f-4bf1f6d4318d" path="/var/lib/kubelet/pods/50e150fc-db48-4659-9c2f-4bf1f6d4318d/volumes" Oct 30 19:14:57.578814 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:57.578704 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:58.971045 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:58.971007 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:59.640511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:59.640472 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk"] Oct 30 19:14:59.653533 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:59.653504 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77"] Oct 30 19:14:59.748640 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:14:59.748609 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:00.187223 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187187 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w"] Oct 30 19:15:00.187692 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187677 4582 cpu_manager.go:414] "containerMap: removing container" podUID="eb8907ec-250b-4c8e-aae5-2274127baa43" containerName="maas-api" Oct 30 19:15:00.187734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187695 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8907ec-250b-4c8e-aae5-2274127baa43" containerName="maas-api" Oct 30 19:15:00.187734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187710 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" Oct 30 19:15:00.187734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187715 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" Oct 30 19:15:00.187734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187723 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8956a23d-6d46-4674-8fc2-6d339cc20c6d" containerName="maas-api" Oct 30 19:15:00.187734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187727 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8956a23d-6d46-4674-8fc2-6d339cc20c6d" containerName="maas-api" Oct 30 19:15:00.187734 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187734 4582 cpu_manager.go:414] "containerMap: removing container" podUID="94045cce-08d4-4f95-8af7-5dd271d8ef96" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187738 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94045cce-08d4-4f95-8af7-5dd271d8ef96" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187745 4582 cpu_manager.go:414] "containerMap: removing container" podUID="d57db115-c40a-49da-bc23-bb87da91713d" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187750 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57db115-c40a-49da-bc23-bb87da91713d" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187756 4582 cpu_manager.go:414] "containerMap: removing container" podUID="ce301cb9-9812-4f86-abd9-b1d41e8833ea" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187761 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce301cb9-9812-4f86-abd9-b1d41e8833ea" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187773 4582 cpu_manager.go:414] "containerMap: removing container" podUID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" containerName="payload-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187779 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" containerName="payload-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187790 4582 cpu_manager.go:414] "containerMap: removing container" podUID="50e150fc-db48-4659-9c2f-4bf1f6d4318d" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187796 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e150fc-db48-4659-9c2f-4bf1f6d4318d" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187830 4582 cpu_manager.go:414] "containerMap: removing container" podUID="4053cb6c-e305-435c-bd77-8575aa05d772" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187836 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4053cb6c-e305-435c-bd77-8575aa05d772" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187856 4582 cpu_manager.go:414] "containerMap: removing container" podUID="601ad845-dc26-4eb2-be78-4ef20d119b09" containerName="main" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187862 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601ad845-dc26-4eb2-be78-4ef20d119b09" containerName="main" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187870 4582 cpu_manager.go:414] "containerMap: removing container" podUID="b2c87fb8-c070-4a6d-9aad-29237f31941b" containerName="payload-pre-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187875 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c87fb8-c070-4a6d-9aad-29237f31941b" containerName="payload-pre-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187883 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7835c137-0861-48e2-bb3f-ea72ef3060da" containerName="main" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187887 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7835c137-0861-48e2-bb3f-ea72ef3060da" containerName="main" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187897 4582 cpu_manager.go:414] "containerMap: removing container" podUID="96bab029-b5b1-40d4-bb1d-df4a3461ee67" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187903 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bab029-b5b1-40d4-bb1d-df4a3461ee67" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187914 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187920 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187930 4582 cpu_manager.go:414] "containerMap: removing container" podUID="1f410e59-f1d4-4741-9d50-ba48d7b67588" containerName="payload-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187935 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f410e59-f1d4-4741-9d50-ba48d7b67588" containerName="payload-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187941 4582 cpu_manager.go:414] "containerMap: removing container" podUID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" containerName="payload-pre-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187947 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" containerName="payload-pre-processing" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187955 4582 cpu_manager.go:414] "containerMap: removing container" podUID="21a928e3-74fc-442e-a69d-794dceb6774c" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187960 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a928e3-74fc-442e-a69d-794dceb6774c" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187966 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187971 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187985 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9997e625-9d15-4297-8813-cbde3f862c8b" containerName="maas-api" Oct 30 19:15:00.187970 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187990 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9997e625-9d15-4297-8813-cbde3f862c8b" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.187998 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3b46eba5-ffed-4e96-ad32-dec766983bd2" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188005 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b46eba5-ffed-4e96-ad32-dec766983bd2" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188016 4582 cpu_manager.go:414] "containerMap: removing container" podUID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188021 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188029 4582 cpu_manager.go:414] "containerMap: removing container" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188033 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188038 4582 cpu_manager.go:414] "containerMap: removing container" podUID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188042 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188050 4582 cpu_manager.go:414] "containerMap: removing container" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188055 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188062 4582 cpu_manager.go:414] "containerMap: removing container" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188066 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188071 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cc76d911-27d1-4fa7-b986-47ccdb9b6c43" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188077 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc76d911-27d1-4fa7-b986-47ccdb9b6c43" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188088 4582 cpu_manager.go:414] "containerMap: removing container" podUID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188093 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188103 4582 cpu_manager.go:414] "containerMap: removing container" podUID="cad19fb9-c8d7-48f0-9d05-992cafb88bbc" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188107 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad19fb9-c8d7-48f0-9d05-992cafb88bbc" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188113 4582 cpu_manager.go:414] "containerMap: removing container" podUID="fa546d21-3cfb-473f-a3fd-470fa68ad607" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188118 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa546d21-3cfb-473f-a3fd-470fa68ad607" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188124 4582 cpu_manager.go:414] "containerMap: removing container" podUID="19653dd4-9014-4a62-a060-97a9dd1e6ba9" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188128 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19653dd4-9014-4a62-a060-97a9dd1e6ba9" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188201 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="19653dd4-9014-4a62-a060-97a9dd1e6ba9" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188211 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8907ec-250b-4c8e-aae5-2274127baa43" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188218 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d167647-e7ba-4ded-845e-e0737006aaab" containerName="istio-proxy" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188223 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8956a23d-6d46-4674-8fc2-6d339cc20c6d" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188229 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="94045cce-08d4-4f95-8af7-5dd271d8ef96" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188237 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57db115-c40a-49da-bc23-bb87da91713d" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188247 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce301cb9-9812-4f86-abd9-b1d41e8833ea" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188253 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="824ccad0-a794-4ae6-ab9c-b5cf75fda61f" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188261 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e150fc-db48-4659-9c2f-4bf1f6d4318d" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188266 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="4053cb6c-e305-435c-bd77-8575aa05d772" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188272 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="601ad845-dc26-4eb2-be78-4ef20d119b09" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188279 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c87fb8-c070-4a6d-9aad-29237f31941b" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188285 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7835c137-0861-48e2-bb3f-ea72ef3060da" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188292 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bab029-b5b1-40d4-bb1d-df4a3461ee67" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188298 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf04596b-f187-4b45-b528-2e6698a13711" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188303 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f410e59-f1d4-4741-9d50-ba48d7b67588" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188311 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb32af3-0f8a-45e8-8e2b-3ffdd888c9cf" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188319 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a928e3-74fc-442e-a69d-794dceb6774c" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188327 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6125b7-5d01-4ae2-aaa9-aa7746ab2a89" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188332 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9997e625-9d15-4297-8813-cbde3f862c8b" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188339 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b46eba5-ffed-4e96-ad32-dec766983bd2" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188345 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7b9c7a-5c24-4a9b-aba9-0bc677a73a3e" containerName="payload-pre-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188351 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="3260d1a7-a604-4bc2-b904-a2fe7129bd23" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188356 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab77db3-946a-49b1-bfa1-444f9ecd00ab" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188362 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4187a2-d01f-487d-8ebe-5c835efc4eee" containerName="istio-proxy" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188369 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="feccd1bd-7683-45ee-bbe1-4d20d06491d1" containerName="main" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188375 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc76d911-27d1-4fa7-b986-47ccdb9b6c43" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188380 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5936235-bcfc-4c16-949c-b7e5a1b6fb35" containerName="payload-processing" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188385 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad19fb9-c8d7-48f0-9d05-992cafb88bbc" containerName="maas-api" Oct 30 19:15:00.188911 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.188393 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa546d21-3cfb-473f-a3fd-470fa68ad607" containerName="payload-processing" Oct 30 19:15:00.193098 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.193081 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.234433 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.234402 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:15:00.235640 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.235617 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:00.235910 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.235891 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:00.236597 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.236570 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w"] Oct 30 19:15:00.238522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.238505 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh"] Oct 30 19:15:00.242123 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.242108 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx"] Oct 30 19:15:00.242244 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.242226 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.247028 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.246999 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.255891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.255868 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx"] Oct 30 19:15:00.273316 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.273293 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh"] Oct 30 19:15:00.277050 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.277022 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jwl\" (UniqueName: \"kubernetes.io/projected/5cbadbcc-562f-40be-846a-c21840a16eac-kube-api-access-g9jwl\") pod \"maas-api-key-cleanup-29889795-qhklx\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.277156 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.277080 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh7l7\" (UniqueName: \"kubernetes.io/projected/16d1d4fc-e4d1-4a19-9b20-beea70199eea-kube-api-access-nh7l7\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.277240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.277189 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbadbcc-562f-40be-846a-c21840a16eac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-qhklx\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.277240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.277225 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05304f3-6bf0-4034-9961-58878f20c73d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.277356 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.277291 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8v5\" (UniqueName: \"kubernetes.io/projected/c05304f3-6bf0-4034-9961-58878f20c73d-kube-api-access-5r8v5\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.277416 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.277354 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d1d4fc-e4d1-4a19-9b20-beea70199eea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.378751 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.378718 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d1d4fc-e4d1-4a19-9b20-beea70199eea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.378952 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.378765 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jwl\" (UniqueName: \"kubernetes.io/projected/5cbadbcc-562f-40be-846a-c21840a16eac-kube-api-access-g9jwl\") pod \"maas-api-key-cleanup-29889795-qhklx\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.378952 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.378860 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh7l7\" (UniqueName: \"kubernetes.io/projected/16d1d4fc-e4d1-4a19-9b20-beea70199eea-kube-api-access-nh7l7\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.378952 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.378949 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbadbcc-562f-40be-846a-c21840a16eac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-qhklx\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.379174 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.378979 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05304f3-6bf0-4034-9961-58878f20c73d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.379174 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.379132 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8v5\" (UniqueName: \"kubernetes.io/projected/c05304f3-6bf0-4034-9961-58878f20c73d-kube-api-access-5r8v5\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.379480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.379457 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d1d4fc-e4d1-4a19-9b20-beea70199eea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.379613 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.379594 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbadbcc-562f-40be-846a-c21840a16eac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-qhklx\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.379653 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.379625 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05304f3-6bf0-4034-9961-58878f20c73d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.399273 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.399248 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8v5\" (UniqueName: \"kubernetes.io/projected/c05304f3-6bf0-4034-9961-58878f20c73d-kube-api-access-5r8v5\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.399403 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.399261 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh7l7\" (UniqueName: \"kubernetes.io/projected/16d1d4fc-e4d1-4a19-9b20-beea70199eea-kube-api-access-nh7l7\") pod \"maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.399403 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.399380 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jwl\" (UniqueName: \"kubernetes.io/projected/5cbadbcc-562f-40be-846a-c21840a16eac-kube-api-access-g9jwl\") pod \"maas-api-key-cleanup-29889795-qhklx\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.506239 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.506169 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:00.562636 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.562445 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:00.571669 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.571638 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:00.687696 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.687670 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 19:15:00.783559 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.783480 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ff8650-d22d-4c6f-a351-fcb70e40980f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff8650-d22d-4c6f-a351-fcb70e40980f-plugins-config-volume\") pod \"66ff8650-d22d-4c6f-a351-fcb70e40980f\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " Oct 30 19:15:00.783705 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.783607 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ff8650-d22d-4c6f-a351-fcb70e40980f-kube-api-access-4gfsn\" (UniqueName: \"kubernetes.io/projected/66ff8650-d22d-4c6f-a351-fcb70e40980f-kube-api-access-4gfsn\") pod \"66ff8650-d22d-4c6f-a351-fcb70e40980f\" (UID: \"66ff8650-d22d-4c6f-a351-fcb70e40980f\") " Oct 30 19:15:00.783878 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.783858 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ff8650-d22d-4c6f-a351-fcb70e40980f-plugins-config-volume" pod "66ff8650-d22d-4c6f-a351-fcb70e40980f" (UID: "66ff8650-d22d-4c6f-a351-fcb70e40980f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:00.783996 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.783981 4582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ff8650-d22d-4c6f-a351-fcb70e40980f-plugins-config-volume\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:00.785671 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.785648 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ff8650-d22d-4c6f-a351-fcb70e40980f-kube-api-access-4gfsn" pod "66ff8650-d22d-4c6f-a351-fcb70e40980f" (UID: "66ff8650-d22d-4c6f-a351-fcb70e40980f"). InnerVolumeSpecName "kube-api-access-4gfsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:00.798001 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.797966 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:00.804330 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.804308 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w"] Oct 30 19:15:00.806399 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:15:00.806375 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05304f3_6bf0_4034_9961_58878f20c73d.slice/crio-58561627c14a3f6788e2674a80d20b660664838eef4580a0cd91a722ac2c9a06 WatchSource:0}: Error finding container 58561627c14a3f6788e2674a80d20b660664838eef4580a0cd91a722ac2c9a06: Status 404 returned error can't find the container with id 58561627c14a3f6788e2674a80d20b660664838eef4580a0cd91a722ac2c9a06 Oct 30 19:15:00.881226 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.881203 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh"] Oct 30 19:15:00.883355 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:15:00.883328 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d1d4fc_e4d1_4a19_9b20_beea70199eea.slice/crio-de19e383f8173810c9c02e1b37ce17016f27ffa0534d604de5c605a028264ece WatchSource:0}: Error finding container de19e383f8173810c9c02e1b37ce17016f27ffa0534d604de5c605a028264ece: Status 404 returned error can't find the container with id de19e383f8173810c9c02e1b37ce17016f27ffa0534d604de5c605a028264ece Oct 30 19:15:00.884892 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.884843 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfsn\" (UniqueName: \"kubernetes.io/projected/66ff8650-d22d-4c6f-a351-fcb70e40980f-kube-api-access-4gfsn\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:00.887433 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.887405 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx"] Oct 30 19:15:00.889248 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:15:00.889215 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbadbcc_562f_40be_846a_c21840a16eac.slice/crio-14cd12cbb052db780576c895c3ac7aa14f5d98989e6dfda5c5929f1090c421d9 WatchSource:0}: Error finding container 14cd12cbb052db780576c895c3ac7aa14f5d98989e6dfda5c5929f1090c421d9: Status 404 returned error can't find the container with id 14cd12cbb052db780576c895c3ac7aa14f5d98989e6dfda5c5929f1090c421d9 Oct 30 19:15:00.979971 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.979938 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" event={"ID":"c05304f3-6bf0-4034-9961-58878f20c73d","Type":"ContainerStarted","Data":"58561627c14a3f6788e2674a80d20b660664838eef4580a0cd91a722ac2c9a06"} Oct 30 19:15:00.981139 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.981117 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" event={"ID":"16d1d4fc-e4d1-4a19-9b20-beea70199eea","Type":"ContainerStarted","Data":"de19e383f8173810c9c02e1b37ce17016f27ffa0534d604de5c605a028264ece"} Oct 30 19:15:00.982325 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.982304 4582 generic.go:356] "Generic (PLEG): container finished" podID="66ff8650-d22d-4c6f-a351-fcb70e40980f" containerID="f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff" exitCode=0 Oct 30 19:15:00.982419 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.982350 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" Oct 30 19:15:00.982419 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.982396 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" event={"ID":"66ff8650-d22d-4c6f-a351-fcb70e40980f","Type":"ContainerDied","Data":"f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff"} Oct 30 19:15:00.982550 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.982444 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c" event={"ID":"66ff8650-d22d-4c6f-a351-fcb70e40980f","Type":"ContainerDied","Data":"ab5ddddd25445c3fa3dcce8c24bed80b97a97c84f5946e0264f327c154a4dd7e"} Oct 30 19:15:00.982550 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.982474 4582 scope.go:122] "RemoveContainer" containerID="f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff" Oct 30 19:15:00.983803 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.983778 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" event={"ID":"5cbadbcc-562f-40be-846a-c21840a16eac","Type":"ContainerStarted","Data":"14cd12cbb052db780576c895c3ac7aa14f5d98989e6dfda5c5929f1090c421d9"} Oct 30 19:15:00.996972 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.996956 4582 scope.go:122] "RemoveContainer" containerID="f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff" Oct 30 19:15:00.997202 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:00.997182 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff\": container with ID starting with f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff not found: ID does not exist" containerID="f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff" Oct 30 19:15:00.997240 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:00.997210 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff"} err="failed to get container status \"f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff\": rpc error: code = NotFound desc = could not find container \"f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff\": container with ID starting with f9538519efe3c88bf18c702f6d055ece68e499c3a8894445a6d9f16a736cb8ff not found: ID does not exist" Oct 30 19:15:01.034684 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.034630 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c"] Oct 30 19:15:01.037946 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.037921 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-24b124-6454ddcf88-rgb5c"] Oct 30 19:15:01.300174 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.300117 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ff8650-d22d-4c6f-a351-fcb70e40980f" path="/var/lib/kubelet/pods/66ff8650-d22d-4c6f-a351-fcb70e40980f/volumes" Oct 30 19:15:01.992268 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.992235 4582 generic.go:356] "Generic (PLEG): container finished" podID="5cbadbcc-562f-40be-846a-c21840a16eac" containerID="55f69a0f05e2d441de63676446f5ff4cce2301ba1ff57b537c3f5632a7730f27" exitCode=60 Oct 30 19:15:01.992487 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.992320 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" event={"ID":"5cbadbcc-562f-40be-846a-c21840a16eac","Type":"ContainerDied","Data":"55f69a0f05e2d441de63676446f5ff4cce2301ba1ff57b537c3f5632a7730f27"} Oct 30 19:15:01.992626 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.992611 4582 scope.go:122] "RemoveContainer" containerID="55f69a0f05e2d441de63676446f5ff4cce2301ba1ff57b537c3f5632a7730f27" Oct 30 19:15:01.994192 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.994116 4582 generic.go:356] "Generic (PLEG): container finished" podID="c05304f3-6bf0-4034-9961-58878f20c73d" containerID="2d7f6ec9d887c078d151cf499a1415db9475bd929840e967d225cfcc36cd219e" exitCode=60 Oct 30 19:15:01.994281 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.994179 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" event={"ID":"c05304f3-6bf0-4034-9961-58878f20c73d","Type":"ContainerDied","Data":"2d7f6ec9d887c078d151cf499a1415db9475bd929840e967d225cfcc36cd219e"} Oct 30 19:15:01.994492 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.994470 4582 scope.go:122] "RemoveContainer" containerID="2d7f6ec9d887c078d151cf499a1415db9475bd929840e967d225cfcc36cd219e" Oct 30 19:15:01.995974 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.995953 4582 generic.go:356] "Generic (PLEG): container finished" podID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerID="4ae3a2fab4f59ffa195400e370d5e7327ee3853cbbee906ce39b92ae6b6d15a6" exitCode=60 Oct 30 19:15:01.996075 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.996058 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" event={"ID":"16d1d4fc-e4d1-4a19-9b20-beea70199eea","Type":"ContainerDied","Data":"4ae3a2fab4f59ffa195400e370d5e7327ee3853cbbee906ce39b92ae6b6d15a6"} Oct 30 19:15:01.996439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:01.996420 4582 scope.go:122] "RemoveContainer" containerID="4ae3a2fab4f59ffa195400e370d5e7327ee3853cbbee906ce39b92ae6b6d15a6" Oct 30 19:15:02.647093 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:02.647051 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:03.005146 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.005061 4582 generic.go:356] "Generic (PLEG): container finished" podID="5cbadbcc-562f-40be-846a-c21840a16eac" containerID="3c16f73f207c135972c2cb5fb6beea76e35ac1e4f019902d82da155736881e15" exitCode=60 Oct 30 19:15:03.005313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.005147 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" event={"ID":"5cbadbcc-562f-40be-846a-c21840a16eac","Type":"ContainerDied","Data":"3c16f73f207c135972c2cb5fb6beea76e35ac1e4f019902d82da155736881e15"} Oct 30 19:15:03.005313 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.005233 4582 scope.go:122] "RemoveContainer" containerID="55f69a0f05e2d441de63676446f5ff4cce2301ba1ff57b537c3f5632a7730f27" Oct 30 19:15:03.005478 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.005461 4582 scope.go:122] "RemoveContainer" containerID="3c16f73f207c135972c2cb5fb6beea76e35ac1e4f019902d82da155736881e15" Oct 30 19:15:03.005780 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:03.005753 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-qhklx_odh-ai-gateway-infra(5cbadbcc-562f-40be-846a-c21840a16eac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" Oct 30 19:15:03.007211 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.007147 4582 generic.go:356] "Generic (PLEG): container finished" podID="c05304f3-6bf0-4034-9961-58878f20c73d" containerID="a4e49b16093441a9160fe39190f061c4fd4a0ad933a42d2e81bdee6dfa1521b1" exitCode=60 Oct 30 19:15:03.007328 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.007274 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" event={"ID":"c05304f3-6bf0-4034-9961-58878f20c73d","Type":"ContainerDied","Data":"a4e49b16093441a9160fe39190f061c4fd4a0ad933a42d2e81bdee6dfa1521b1"} Oct 30 19:15:03.007601 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.007584 4582 scope.go:122] "RemoveContainer" containerID="a4e49b16093441a9160fe39190f061c4fd4a0ad933a42d2e81bdee6dfa1521b1" Oct 30 19:15:03.007855 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:03.007809 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w_odh-ai-gateway-infra(c05304f3-6bf0-4034-9961-58878f20c73d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" Oct 30 19:15:03.009168 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.009151 4582 generic.go:356] "Generic (PLEG): container finished" podID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerID="3bb38d9479623c40633ddd84b10f31e2f33d14c583f8d92df7b4e29af9739659" exitCode=60 Oct 30 19:15:03.009242 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.009199 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" event={"ID":"16d1d4fc-e4d1-4a19-9b20-beea70199eea","Type":"ContainerDied","Data":"3bb38d9479623c40633ddd84b10f31e2f33d14c583f8d92df7b4e29af9739659"} Oct 30 19:15:03.009396 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.009382 4582 scope.go:122] "RemoveContainer" containerID="3bb38d9479623c40633ddd84b10f31e2f33d14c583f8d92df7b4e29af9739659" Oct 30 19:15:03.009578 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:03.009562 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh_odh-ai-gateway-infra(16d1d4fc-e4d1-4a19-9b20-beea70199eea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" Oct 30 19:15:03.016944 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.016927 4582 scope.go:122] "RemoveContainer" containerID="2d7f6ec9d887c078d151cf499a1415db9475bd929840e967d225cfcc36cd219e" Oct 30 19:15:03.029360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:03.029343 4582 scope.go:122] "RemoveContainer" containerID="4ae3a2fab4f59ffa195400e370d5e7327ee3853cbbee906ce39b92ae6b6d15a6" Oct 30 19:15:04.016383 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.016351 4582 scope.go:122] "RemoveContainer" containerID="3bb38d9479623c40633ddd84b10f31e2f33d14c583f8d92df7b4e29af9739659" Oct 30 19:15:04.016849 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:04.016630 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh_odh-ai-gateway-infra(16d1d4fc-e4d1-4a19-9b20-beea70199eea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" Oct 30 19:15:04.017912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.017891 4582 scope.go:122] "RemoveContainer" containerID="3c16f73f207c135972c2cb5fb6beea76e35ac1e4f019902d82da155736881e15" Oct 30 19:15:04.018087 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:04.018061 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-qhklx_odh-ai-gateway-infra(5cbadbcc-562f-40be-846a-c21840a16eac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" Oct 30 19:15:04.019355 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.019336 4582 scope.go:122] "RemoveContainer" containerID="a4e49b16093441a9160fe39190f061c4fd4a0ad933a42d2e81bdee6dfa1521b1" Oct 30 19:15:04.019508 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:04.019490 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w_odh-ai-gateway-infra(c05304f3-6bf0-4034-9961-58878f20c73d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" Oct 30 19:15:04.041685 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.041662 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:04.142665 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.142631 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:04.433117 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.433087 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:04.545426 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.545392 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:04.866858 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:04.866762 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:05.146271 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.146234 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj"] Oct 30 19:15:05.146732 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.146718 4582 cpu_manager.go:414] "containerMap: removing container" podUID="66ff8650-d22d-4c6f-a351-fcb70e40980f" containerName="payload-processing" Oct 30 19:15:05.146777 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.146733 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ff8650-d22d-4c6f-a351-fcb70e40980f" containerName="payload-processing" Oct 30 19:15:05.146849 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.146838 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ff8650-d22d-4c6f-a351-fcb70e40980f" containerName="payload-processing" Oct 30 19:15:05.151341 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.151322 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.168140 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.168115 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj"] Oct 30 19:15:05.177284 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.177262 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:05.180416 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.180390 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:05.229634 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.229609 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.229765 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.229744 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqsgw\" (UniqueName: \"kubernetes.io/projected/0b1fe9f0-55d7-4818-ab9a-9516f2820145-kube-api-access-wqsgw\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.229886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.229871 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.285584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.285551 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:05.331110 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.331082 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqsgw\" (UniqueName: \"kubernetes.io/projected/0b1fe9f0-55d7-4818-ab9a-9516f2820145-kube-api-access-wqsgw\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.331248 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.331156 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.331248 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.331182 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.331887 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.331865 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.333559 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.333538 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.350415 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.350389 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqsgw\" (UniqueName: \"kubernetes.io/projected/0b1fe9f0-55d7-4818-ab9a-9516f2820145-kube-api-access-wqsgw\") pod \"maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.464501 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.464429 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:05.533068 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.530244 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:05.769296 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.769272 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj"] Oct 30 19:15:05.771644 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:15:05.771619 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1fe9f0_55d7_4818_ab9a_9516f2820145.slice/crio-9781fc4a57d1be66c39b5256083301362e88ab0bd00bd9254b3a31d40a54115a WatchSource:0}: Error finding container 9781fc4a57d1be66c39b5256083301362e88ab0bd00bd9254b3a31d40a54115a: Status 404 returned error can't find the container with id 9781fc4a57d1be66c39b5256083301362e88ab0bd00bd9254b3a31d40a54115a Oct 30 19:15:05.979870 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:05.979838 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:06.031473 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:06.031408 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" event={"ID":"0b1fe9f0-55d7-4818-ab9a-9516f2820145","Type":"ContainerStarted","Data":"9781fc4a57d1be66c39b5256083301362e88ab0bd00bd9254b3a31d40a54115a"} Oct 30 19:15:06.224991 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:06.224956 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:15:06.556322 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:06.556288 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:07.038455 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:07.038423 4582 generic.go:356] "Generic (PLEG): container finished" podID="0b1fe9f0-55d7-4818-ab9a-9516f2820145" containerID="547886e223d9d26dd477262582783237de871af25538243fd5ff412afa6a266b" exitCode=0 Oct 30 19:15:07.038654 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:07.038507 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" event={"ID":"0b1fe9f0-55d7-4818-ab9a-9516f2820145","Type":"ContainerDied","Data":"547886e223d9d26dd477262582783237de871af25538243fd5ff412afa6a266b"} Oct 30 19:15:08.192789 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.192765 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:08.260894 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.260863 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1fe9f0-55d7-4818-ab9a-9516f2820145-kube-api-access-wqsgw\" (UniqueName: \"kubernetes.io/projected/0b1fe9f0-55d7-4818-ab9a-9516f2820145-kube-api-access-wqsgw\") pod \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " Oct 30 19:15:08.261053 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.260913 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-xks-ca\") pod \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " Oct 30 19:15:08.261053 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.260977 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-openshift-ca\") pod \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\" (UID: \"0b1fe9f0-55d7-4818-ab9a-9516f2820145\") " Oct 30 19:15:08.261404 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.261378 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-openshift-ca" pod "0b1fe9f0-55d7-4818-ab9a-9516f2820145" (UID: "0b1fe9f0-55d7-4818-ab9a-9516f2820145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:08.262998 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.262974 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1fe9f0-55d7-4818-ab9a-9516f2820145-kube-api-access-wqsgw" pod "0b1fe9f0-55d7-4818-ab9a-9516f2820145" (UID: "0b1fe9f0-55d7-4818-ab9a-9516f2820145"). InnerVolumeSpecName "kube-api-access-wqsgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:08.263157 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.263137 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-xks-ca" pod "0b1fe9f0-55d7-4818-ab9a-9516f2820145" (UID: "0b1fe9f0-55d7-4818-ab9a-9516f2820145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:08.290167 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.290143 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:08.363342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.363258 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:08.363342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.363288 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqsgw\" (UniqueName: \"kubernetes.io/projected/0b1fe9f0-55d7-4818-ab9a-9516f2820145-kube-api-access-wqsgw\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:08.363342 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.363302 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1fe9f0-55d7-4818-ab9a-9516f2820145-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:08.571563 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.571531 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:08.663849 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:08.663804 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:09.051739 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:09.051664 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" Oct 30 19:15:09.051895 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:09.051664 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj" event={"ID":"0b1fe9f0-55d7-4818-ab9a-9516f2820145","Type":"ContainerDied","Data":"9781fc4a57d1be66c39b5256083301362e88ab0bd00bd9254b3a31d40a54115a"} Oct 30 19:15:09.051895 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:09.051789 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9781fc4a57d1be66c39b5256083301362e88ab0bd00bd9254b3a31d40a54115a" Oct 30 19:15:09.911666 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:09.911635 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4"] Oct 30 19:15:10.005044 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:10.005007 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:10.105026 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:10.104981 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:10.963279 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:10.963238 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7"] Oct 30 19:15:11.037572 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:11.037538 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:11.973888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:11.973856 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:13.864776 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:13.864738 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:14.292930 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:14.292900 4582 scope.go:122] "RemoveContainer" containerID="3bb38d9479623c40633ddd84b10f31e2f33d14c583f8d92df7b4e29af9739659" Oct 30 19:15:14.662681 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:14.662638 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" containerID="cri-o://d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c" gracePeriod=45 Oct 30 19:15:14.675239 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:14.675205 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" containerID="cri-o://1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1" gracePeriod=45 Oct 30 19:15:14.711936 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:14.711907 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused" Oct 30 19:15:14.768766 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:14.768732 4582 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Oct 30 19:15:14.938357 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:14.938326 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 19:15:14.951491 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:14.951467 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 19:15:15.030620 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030585 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-model-cache\") pod \"0d0c1119-b013-48be-8604-a21ed73bf11e\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " Oct 30 19:15:15.030797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030629 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f23fe3c-a468-4011-a210-aabb09c25695-kube-api-access-xt45b\" (UniqueName: \"kubernetes.io/projected/7f23fe3c-a468-4011-a210-aabb09c25695-kube-api-access-xt45b\") pod \"7f23fe3c-a468-4011-a210-aabb09c25695\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " Oct 30 19:15:15.030797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030656 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-dshm\") pod \"7f23fe3c-a468-4011-a210-aabb09c25695\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " Oct 30 19:15:15.030797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030688 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0c1119-b013-48be-8604-a21ed73bf11e-kube-api-access-5hpn8\" (UniqueName: \"kubernetes.io/projected/0d0c1119-b013-48be-8604-a21ed73bf11e-kube-api-access-5hpn8\") pod \"0d0c1119-b013-48be-8604-a21ed73bf11e\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " Oct 30 19:15:15.030797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030731 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-dshm\") pod \"0d0c1119-b013-48be-8604-a21ed73bf11e\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " Oct 30 19:15:15.030797 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030766 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0c1119-b013-48be-8604-a21ed73bf11e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0c1119-b013-48be-8604-a21ed73bf11e-tls-certs\") pod \"0d0c1119-b013-48be-8604-a21ed73bf11e\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " Oct 30 19:15:15.031134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030803 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f23fe3c-a468-4011-a210-aabb09c25695-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f23fe3c-a468-4011-a210-aabb09c25695-tls-certs\") pod \"7f23fe3c-a468-4011-a210-aabb09c25695\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " Oct 30 19:15:15.031134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030867 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-model-cache\") pod \"7f23fe3c-a468-4011-a210-aabb09c25695\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " Oct 30 19:15:15.031134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030900 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-home\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-home\") pod \"7f23fe3c-a468-4011-a210-aabb09c25695\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " Oct 30 19:15:15.031134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030914 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-model-cache" pod "0d0c1119-b013-48be-8604-a21ed73bf11e" (UID: "0d0c1119-b013-48be-8604-a21ed73bf11e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.031134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.030966 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-home\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-home\") pod \"0d0c1119-b013-48be-8604-a21ed73bf11e\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " Oct 30 19:15:15.031134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031021 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-tmp-dir\") pod \"7f23fe3c-a468-4011-a210-aabb09c25695\" (UID: \"7f23fe3c-a468-4011-a210-aabb09c25695\") " Oct 30 19:15:15.031134 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031068 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-tmp-dir\") pod \"0d0c1119-b013-48be-8604-a21ed73bf11e\" (UID: \"0d0c1119-b013-48be-8604-a21ed73bf11e\") " Oct 30 19:15:15.031548 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031432 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-home" pod "0d0c1119-b013-48be-8604-a21ed73bf11e" (UID: "0d0c1119-b013-48be-8604-a21ed73bf11e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.031548 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031537 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.031663 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031557 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.031791 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031620 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-home" pod "7f23fe3c-a468-4011-a210-aabb09c25695" (UID: "7f23fe3c-a468-4011-a210-aabb09c25695"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.031891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031625 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-tmp-dir" pod "0d0c1119-b013-48be-8604-a21ed73bf11e" (UID: "0d0c1119-b013-48be-8604-a21ed73bf11e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.031891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.031794 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-tmp-dir" pod "7f23fe3c-a468-4011-a210-aabb09c25695" (UID: "7f23fe3c-a468-4011-a210-aabb09c25695"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.032127 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.032106 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-model-cache" pod "7f23fe3c-a468-4011-a210-aabb09c25695" (UID: "7f23fe3c-a468-4011-a210-aabb09c25695"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.033290 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.033266 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c1119-b013-48be-8604-a21ed73bf11e-tls-certs" pod "0d0c1119-b013-48be-8604-a21ed73bf11e" (UID: "0d0c1119-b013-48be-8604-a21ed73bf11e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:15.033463 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.033440 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-dshm" pod "7f23fe3c-a468-4011-a210-aabb09c25695" (UID: "7f23fe3c-a468-4011-a210-aabb09c25695"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.035181 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.035131 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f23fe3c-a468-4011-a210-aabb09c25695-kube-api-access-xt45b" pod "7f23fe3c-a468-4011-a210-aabb09c25695" (UID: "7f23fe3c-a468-4011-a210-aabb09c25695"). InnerVolumeSpecName "kube-api-access-xt45b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:15.035334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.035269 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0c1119-b013-48be-8604-a21ed73bf11e-kube-api-access-5hpn8" pod "0d0c1119-b013-48be-8604-a21ed73bf11e" (UID: "0d0c1119-b013-48be-8604-a21ed73bf11e"). InnerVolumeSpecName "kube-api-access-5hpn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:15.035334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.035282 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-dshm" pod "0d0c1119-b013-48be-8604-a21ed73bf11e" (UID: "0d0c1119-b013-48be-8604-a21ed73bf11e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.035334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.035299 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f23fe3c-a468-4011-a210-aabb09c25695-tls-certs" pod "7f23fe3c-a468-4011-a210-aabb09c25695" (UID: "7f23fe3c-a468-4011-a210-aabb09c25695"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:15.086018 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.085981 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" event={"ID":"16d1d4fc-e4d1-4a19-9b20-beea70199eea","Type":"ContainerStarted","Data":"de6a680deff4bca0d8faa376df9ad722e79cd160fda1b00eec9257685cfd7df5"} Oct 30 19:15:15.087333 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.087307 4582 generic.go:356] "Generic (PLEG): container finished" podID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerID="d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c" exitCode=0 Oct 30 19:15:15.087480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.087346 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" Oct 30 19:15:15.087480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.087391 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" event={"ID":"0d0c1119-b013-48be-8604-a21ed73bf11e","Type":"ContainerDied","Data":"d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c"} Oct 30 19:15:15.087480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.087436 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk" event={"ID":"0d0c1119-b013-48be-8604-a21ed73bf11e","Type":"ContainerDied","Data":"99a01754477d31ac06b55e702efcf29963fae96acd6911a79c39a559a98ebc14"} Oct 30 19:15:15.087480 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.087459 4582 scope.go:122] "RemoveContainer" containerID="d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c" Oct 30 19:15:15.088786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.088765 4582 generic.go:356] "Generic (PLEG): container finished" podID="7f23fe3c-a468-4011-a210-aabb09c25695" containerID="1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1" exitCode=0 Oct 30 19:15:15.088904 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.088809 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" Oct 30 19:15:15.088975 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.088815 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" event={"ID":"7f23fe3c-a468-4011-a210-aabb09c25695","Type":"ContainerDied","Data":"1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1"} Oct 30 19:15:15.088975 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.088929 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77" event={"ID":"7f23fe3c-a468-4011-a210-aabb09c25695","Type":"ContainerDied","Data":"7c45736bbee7cc6da4e0dd853986a0fa68223a5d7bcc8c89e061f5b86d27712b"} Oct 30 19:15:15.097694 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.097667 4582 scope.go:122] "RemoveContainer" containerID="d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c" Oct 30 19:15:15.097979 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:15.097957 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c\": container with ID starting with d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c not found: ID does not exist" containerID="d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c" Oct 30 19:15:15.098066 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.097987 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c"} err="failed to get container status \"d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c\": rpc error: code = NotFound desc = could not find container \"d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c\": container with ID starting with d70b8b418d02f4f497f920abf984c91f9d5233e19a3a598ca16d6ba9da545f5c not found: ID does not exist" Oct 30 19:15:15.098066 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.098003 4582 scope.go:122] "RemoveContainer" containerID="1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1" Oct 30 19:15:15.106038 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.106021 4582 scope.go:122] "RemoveContainer" containerID="1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1" Oct 30 19:15:15.106292 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:15.106273 4582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1\": container with ID starting with 1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1 not found: ID does not exist" containerID="1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1" Oct 30 19:15:15.106338 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.106300 4582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1"} err="failed to get container status \"1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1\": rpc error: code = NotFound desc = could not find container \"1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1\": container with ID starting with 1dc44a86a96c20c4efff613616c0aa472da3e44d0aff6283ab0ca776777d93e1 not found: ID does not exist" Oct 30 19:15:15.124325 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.124276 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" podStartSLOduration=15.124262704 podStartE2EDuration="15.124262704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:15.123240687 +0000 UTC m=+4611.947634980" watchObservedRunningTime="2026-10-30 19:15:15.124262704 +0000 UTC m=+4611.948656996" Oct 30 19:15:15.132351 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132324 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132351 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132350 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132360 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132369 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132378 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt45b\" (UniqueName: \"kubernetes.io/projected/7f23fe3c-a468-4011-a210-aabb09c25695-kube-api-access-xt45b\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132387 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f23fe3c-a468-4011-a210-aabb09c25695-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132395 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hpn8\" (UniqueName: \"kubernetes.io/projected/0d0c1119-b013-48be-8604-a21ed73bf11e-kube-api-access-5hpn8\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132403 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d0c1119-b013-48be-8604-a21ed73bf11e-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132411 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0c1119-b013-48be-8604-a21ed73bf11e-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.132520 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.132418 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f23fe3c-a468-4011-a210-aabb09c25695-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.158250 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.158227 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk"] Oct 30 19:15:15.161810 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.161786 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-w4-fe1374-kserve-56568bfdc-pwmwk"] Oct 30 19:15:15.194484 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.194461 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77"] Oct 30 19:15:15.198668 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.198647 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/premium-w4-fe1374-kserve-66fd9bb5b8-psv77"] Oct 30 19:15:15.293295 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.293266 4582 scope.go:122] "RemoveContainer" containerID="a4e49b16093441a9160fe39190f061c4fd4a0ad933a42d2e81bdee6dfa1521b1" Oct 30 19:15:15.301491 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.301461 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" path="/var/lib/kubelet/pods/0d0c1119-b013-48be-8604-a21ed73bf11e/volumes" Oct 30 19:15:15.301905 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.301876 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" path="/var/lib/kubelet/pods/7f23fe3c-a468-4011-a210-aabb09c25695/volumes" Oct 30 19:15:15.348318 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348266 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr"] Oct 30 19:15:15.348749 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348733 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" Oct 30 19:15:15.348803 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348751 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" Oct 30 19:15:15.348803 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348771 4582 cpu_manager.go:414] "containerMap: removing container" podUID="0b1fe9f0-55d7-4818-ab9a-9516f2820145" containerName="revoke-keys" Oct 30 19:15:15.348803 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348777 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1fe9f0-55d7-4818-ab9a-9516f2820145" containerName="revoke-keys" Oct 30 19:15:15.348927 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348814 4582 cpu_manager.go:414] "containerMap: removing container" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" Oct 30 19:15:15.348927 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348832 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" Oct 30 19:15:15.348927 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348903 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f23fe3c-a468-4011-a210-aabb09c25695" containerName="main" Oct 30 19:15:15.348927 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348912 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0c1119-b013-48be-8604-a21ed73bf11e" containerName="main" Oct 30 19:15:15.348927 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.348919 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1fe9f0-55d7-4818-ab9a-9516f2820145" containerName="revoke-keys" Oct 30 19:15:15.353230 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.353215 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.369654 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.369627 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr"] Oct 30 19:15:15.435250 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.435216 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.435427 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.435280 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.435498 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.435441 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwt5q\" (UniqueName: \"kubernetes.io/projected/bcff78ab-c780-4e1b-8cde-89e20217a12c-kube-api-access-nwt5q\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.462684 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.462655 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:15.514387 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.514358 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:15.536572 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.536543 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwt5q\" (UniqueName: \"kubernetes.io/projected/bcff78ab-c780-4e1b-8cde-89e20217a12c-kube-api-access-nwt5q\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.536703 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.536638 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.536703 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.536672 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.537337 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.537312 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.538870 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.538850 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.555536 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.555515 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwt5q\" (UniqueName: \"kubernetes.io/projected/bcff78ab-c780-4e1b-8cde-89e20217a12c-kube-api-access-nwt5q\") pod \"maas-api-revoke-sub-7505801d-q82xr\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.618026 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.617999 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:15.671269 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.671244 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:15.760089 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.760052 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:15.969716 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:15.969687 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr"] Oct 30 19:15:15.971514 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:15:15.971486 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcff78ab_c780_4e1b_8cde_89e20217a12c.slice/crio-49545f18c781a5948708356c4a8380cab58cb393b6ccbe293957b26e34233926 WatchSource:0}: Error finding container 49545f18c781a5948708356c4a8380cab58cb393b6ccbe293957b26e34233926: Status 404 returned error can't find the container with id 49545f18c781a5948708356c4a8380cab58cb393b6ccbe293957b26e34233926 Oct 30 19:15:16.095539 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.095506 4582 generic.go:356] "Generic (PLEG): container finished" podID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerID="de6a680deff4bca0d8faa376df9ad722e79cd160fda1b00eec9257685cfd7df5" exitCode=60 Oct 30 19:15:16.095694 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.095574 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" event={"ID":"16d1d4fc-e4d1-4a19-9b20-beea70199eea","Type":"ContainerDied","Data":"de6a680deff4bca0d8faa376df9ad722e79cd160fda1b00eec9257685cfd7df5"} Oct 30 19:15:16.095694 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.095629 4582 scope.go:122] "RemoveContainer" containerID="3bb38d9479623c40633ddd84b10f31e2f33d14c583f8d92df7b4e29af9739659" Oct 30 19:15:16.096024 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.096008 4582 scope.go:122] "RemoveContainer" containerID="de6a680deff4bca0d8faa376df9ad722e79cd160fda1b00eec9257685cfd7df5" Oct 30 19:15:16.096293 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:16.096268 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh_odh-ai-gateway-infra(16d1d4fc-e4d1-4a19-9b20-beea70199eea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" Oct 30 19:15:16.097847 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.097801 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" event={"ID":"bcff78ab-c780-4e1b-8cde-89e20217a12c","Type":"ContainerStarted","Data":"49545f18c781a5948708356c4a8380cab58cb393b6ccbe293957b26e34233926"} Oct 30 19:15:16.101522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.101501 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" event={"ID":"c05304f3-6bf0-4034-9961-58878f20c73d","Type":"ContainerStarted","Data":"26ec0acd306939ad1b9a279f753fef3bf6b2c95f8b8173dd3f663b589b152c6e"} Oct 30 19:15:16.134351 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.134329 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh"] Oct 30 19:15:16.187906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.187856 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" podStartSLOduration=16.187810223 podStartE2EDuration="16.187810223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:16.185511747 +0000 UTC m=+4613.009906059" watchObservedRunningTime="2026-10-30 19:15:16.187810223 +0000 UTC m=+4613.012204516" Oct 30 19:15:16.304783 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.304729 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:16.401310 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:16.401259 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:17.082199 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.082167 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:17.108266 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.108240 4582 generic.go:356] "Generic (PLEG): container finished" podID="c05304f3-6bf0-4034-9961-58878f20c73d" containerID="26ec0acd306939ad1b9a279f753fef3bf6b2c95f8b8173dd3f663b589b152c6e" exitCode=60 Oct 30 19:15:17.108434 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.108346 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" event={"ID":"c05304f3-6bf0-4034-9961-58878f20c73d","Type":"ContainerDied","Data":"26ec0acd306939ad1b9a279f753fef3bf6b2c95f8b8173dd3f663b589b152c6e"} Oct 30 19:15:17.108488 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.108433 4582 scope.go:122] "RemoveContainer" containerID="a4e49b16093441a9160fe39190f061c4fd4a0ad933a42d2e81bdee6dfa1521b1" Oct 30 19:15:17.108922 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.108897 4582 scope.go:122] "RemoveContainer" containerID="26ec0acd306939ad1b9a279f753fef3bf6b2c95f8b8173dd3f663b589b152c6e" Oct 30 19:15:17.109189 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:17.109165 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w_odh-ai-gateway-infra(c05304f3-6bf0-4034-9961-58878f20c73d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" Oct 30 19:15:17.112153 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.112131 4582 generic.go:356] "Generic (PLEG): container finished" podID="bcff78ab-c780-4e1b-8cde-89e20217a12c" containerID="5b0552dbc7e3e8614641a1694c2bd05f6a442172bc7fb06076f9e760c67dbcd2" exitCode=0 Oct 30 19:15:17.112254 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.112162 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" event={"ID":"bcff78ab-c780-4e1b-8cde-89e20217a12c","Type":"ContainerDied","Data":"5b0552dbc7e3e8614641a1694c2bd05f6a442172bc7fb06076f9e760c67dbcd2"} Oct 30 19:15:17.196862 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.189910 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:17.196862 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.195597 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w"] Oct 30 19:15:17.255141 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.255116 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:17.293394 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.293363 4582 scope.go:122] "RemoveContainer" containerID="3c16f73f207c135972c2cb5fb6beea76e35ac1e4f019902d82da155736881e15" Oct 30 19:15:17.330020 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.329991 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9"] Oct 30 19:15:17.330716 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330693 4582 cpu_manager.go:414] "containerMap: removing container" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.330806 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330719 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.330806 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330734 4582 cpu_manager.go:414] "containerMap: removing container" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.330806 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330742 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.330806 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330776 4582 cpu_manager.go:414] "containerMap: removing container" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.330806 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330786 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.331124 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330966 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.331124 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.330981 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.335897 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.335876 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.351558 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.351536 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9"] Oct 30 19:15:17.356623 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.356602 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d1d4fc-e4d1-4a19-9b20-beea70199eea-kube-api-access-nh7l7\" (UniqueName: \"kubernetes.io/projected/16d1d4fc-e4d1-4a19-9b20-beea70199eea-kube-api-access-nh7l7\") pod \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " Oct 30 19:15:17.356757 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.356667 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d1d4fc-e4d1-4a19-9b20-beea70199eea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d1d4fc-e4d1-4a19-9b20-beea70199eea-maas-api-service-ca\") pod \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\" (UID: \"16d1d4fc-e4d1-4a19-9b20-beea70199eea\") " Oct 30 19:15:17.357110 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.357079 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d1d4fc-e4d1-4a19-9b20-beea70199eea-maas-api-service-ca" pod "16d1d4fc-e4d1-4a19-9b20-beea70199eea" (UID: "16d1d4fc-e4d1-4a19-9b20-beea70199eea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:17.358798 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.358775 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d1d4fc-e4d1-4a19-9b20-beea70199eea-kube-api-access-nh7l7" pod "16d1d4fc-e4d1-4a19-9b20-beea70199eea" (UID: "16d1d4fc-e4d1-4a19-9b20-beea70199eea"). InnerVolumeSpecName "kube-api-access-nh7l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:17.457965 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.457931 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql"] Oct 30 19:15:17.458107 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.458078 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5928k\" (UniqueName: \"kubernetes.io/projected/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-kube-api-access-5928k\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.458190 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.458125 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.458266 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.458221 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.458652 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.458632 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh7l7\" (UniqueName: \"kubernetes.io/projected/16d1d4fc-e4d1-4a19-9b20-beea70199eea-kube-api-access-nh7l7\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:17.458758 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.458659 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d1d4fc-e4d1-4a19-9b20-beea70199eea-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:17.458894 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.458874 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" containerName="cleanup" Oct 30 19:15:17.462882 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.462863 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.475663 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.474360 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql"] Oct 30 19:15:17.559501 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.559468 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5928k\" (UniqueName: \"kubernetes.io/projected/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-kube-api-access-5928k\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.559620 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.559516 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.559620 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.559598 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.559715 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.559629 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.559775 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.559725 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.559892 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.559868 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnb6w\" (UniqueName: \"kubernetes.io/projected/fc146841-dec1-42b0-b593-61c9d0e61b6c-kube-api-access-lnb6w\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.560355 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.560335 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.561900 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.561881 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.577667 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.577645 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5928k\" (UniqueName: \"kubernetes.io/projected/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-kube-api-access-5928k\") pod \"maas-api-revoke-sub-dac36549-fnmr9\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.652029 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.651985 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:17.661265 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.661239 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.661398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.661287 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnb6w\" (UniqueName: \"kubernetes.io/projected/fc146841-dec1-42b0-b593-61c9d0e61b6c-kube-api-access-lnb6w\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.661398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.661375 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.662076 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.662056 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.663985 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.663949 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.680188 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.680159 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnb6w\" (UniqueName: \"kubernetes.io/projected/fc146841-dec1-42b0-b593-61c9d0e61b6c-kube-api-access-lnb6w\") pod \"maas-api-revoke-sub-683ae185-nkdql\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.783912 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.783879 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:17.950060 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:17.950035 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9"] Oct 30 19:15:17.951725 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:15:17.951697 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02c7ed0_c765_49e2_8cc8_0d7636c688e5.slice/crio-655969604aa4134b91f0b5905f6c132cd57d48be8a3f50e7c840381aca30b8e8 WatchSource:0}: Error finding container 655969604aa4134b91f0b5905f6c132cd57d48be8a3f50e7c840381aca30b8e8: Status 404 returned error can't find the container with id 655969604aa4134b91f0b5905f6c132cd57d48be8a3f50e7c840381aca30b8e8 Oct 30 19:15:18.080451 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.080421 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql"] Oct 30 19:15:18.081882 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:15:18.081855 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc146841_dec1_42b0_b593_61c9d0e61b6c.slice/crio-170fd0990eaee4e772c0461437c5afb315248b48947a46194369fcd0306256df WatchSource:0}: Error finding container 170fd0990eaee4e772c0461437c5afb315248b48947a46194369fcd0306256df: Status 404 returned error can't find the container with id 170fd0990eaee4e772c0461437c5afb315248b48947a46194369fcd0306256df Oct 30 19:15:18.118354 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.118328 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" event={"ID":"fc146841-dec1-42b0-b593-61c9d0e61b6c","Type":"ContainerStarted","Data":"170fd0990eaee4e772c0461437c5afb315248b48947a46194369fcd0306256df"} Oct 30 19:15:18.119677 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.119653 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" event={"ID":"c02c7ed0-c765-49e2-8cc8-0d7636c688e5","Type":"ContainerStarted","Data":"655969604aa4134b91f0b5905f6c132cd57d48be8a3f50e7c840381aca30b8e8"} Oct 30 19:15:18.122535 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.122514 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" Oct 30 19:15:18.122639 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.122550 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh" event={"ID":"16d1d4fc-e4d1-4a19-9b20-beea70199eea","Type":"ContainerDied","Data":"de19e383f8173810c9c02e1b37ce17016f27ffa0534d604de5c605a028264ece"} Oct 30 19:15:18.122639 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.122617 4582 scope.go:122] "RemoveContainer" containerID="de6a680deff4bca0d8faa376df9ad722e79cd160fda1b00eec9257685cfd7df5" Oct 30 19:15:18.124888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.124861 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" event={"ID":"5cbadbcc-562f-40be-846a-c21840a16eac","Type":"ContainerStarted","Data":"3a766ff470654838d377e199ea2b5ef083044975de928fad5867f100cd995d5d"} Oct 30 19:15:18.196751 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.196722 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh"] Oct 30 19:15:18.200022 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.199993 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-921450-29889795-8szvh"] Oct 30 19:15:18.237791 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.237734 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" podStartSLOduration=18.237719109 podStartE2EDuration="18.237719109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:18.235241438 +0000 UTC m=+4615.059635743" watchObservedRunningTime="2026-10-30 19:15:18.237719109 +0000 UTC m=+4615.062113400" Oct 30 19:15:18.262436 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.262401 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:18.291648 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.291623 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:18.369358 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.369332 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-xks-ca\") pod \"bcff78ab-c780-4e1b-8cde-89e20217a12c\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " Oct 30 19:15:18.369507 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.369477 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-openshift-ca\") pod \"bcff78ab-c780-4e1b-8cde-89e20217a12c\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " Oct 30 19:15:18.369562 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.369533 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcff78ab-c780-4e1b-8cde-89e20217a12c-kube-api-access-nwt5q\" (UniqueName: \"kubernetes.io/projected/bcff78ab-c780-4e1b-8cde-89e20217a12c-kube-api-access-nwt5q\") pod \"bcff78ab-c780-4e1b-8cde-89e20217a12c\" (UID: \"bcff78ab-c780-4e1b-8cde-89e20217a12c\") " Oct 30 19:15:18.369916 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.369877 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-openshift-ca" pod "bcff78ab-c780-4e1b-8cde-89e20217a12c" (UID: "bcff78ab-c780-4e1b-8cde-89e20217a12c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.442545 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.442509 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-xks-ca" pod "bcff78ab-c780-4e1b-8cde-89e20217a12c" (UID: "bcff78ab-c780-4e1b-8cde-89e20217a12c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:18.442545 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.442528 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcff78ab-c780-4e1b-8cde-89e20217a12c-kube-api-access-nwt5q" pod "bcff78ab-c780-4e1b-8cde-89e20217a12c" (UID: "bcff78ab-c780-4e1b-8cde-89e20217a12c"). InnerVolumeSpecName "kube-api-access-nwt5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.471236 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.471185 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05304f3-6bf0-4034-9961-58878f20c73d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05304f3-6bf0-4034-9961-58878f20c73d-maas-api-service-ca\") pod \"c05304f3-6bf0-4034-9961-58878f20c73d\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " Oct 30 19:15:18.471332 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.471303 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05304f3-6bf0-4034-9961-58878f20c73d-kube-api-access-5r8v5\" (UniqueName: \"kubernetes.io/projected/c05304f3-6bf0-4034-9961-58878f20c73d-kube-api-access-5r8v5\") pod \"c05304f3-6bf0-4034-9961-58878f20c73d\" (UID: \"c05304f3-6bf0-4034-9961-58878f20c73d\") " Oct 30 19:15:18.471536 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.471508 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05304f3-6bf0-4034-9961-58878f20c73d-maas-api-service-ca" pod "c05304f3-6bf0-4034-9961-58878f20c73d" (UID: "c05304f3-6bf0-4034-9961-58878f20c73d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.471658 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.471635 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:18.471658 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.471653 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c05304f3-6bf0-4034-9961-58878f20c73d-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:18.471754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.471668 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcff78ab-c780-4e1b-8cde-89e20217a12c-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:18.471754 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.471682 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwt5q\" (UniqueName: \"kubernetes.io/projected/bcff78ab-c780-4e1b-8cde-89e20217a12c-kube-api-access-nwt5q\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:18.473202 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.473182 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05304f3-6bf0-4034-9961-58878f20c73d-kube-api-access-5r8v5" pod "c05304f3-6bf0-4034-9961-58878f20c73d" (UID: "c05304f3-6bf0-4034-9961-58878f20c73d"). InnerVolumeSpecName "kube-api-access-5r8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.572337 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:18.572262 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r8v5\" (UniqueName: \"kubernetes.io/projected/c05304f3-6bf0-4034-9961-58878f20c73d-kube-api-access-5r8v5\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:19.131709 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.131624 4582 generic.go:356] "Generic (PLEG): container finished" podID="fc146841-dec1-42b0-b593-61c9d0e61b6c" containerID="84333229cb398638bc6d8de19ddd01984972309c0ae03e7b74eddc055fc283eb" exitCode=0 Oct 30 19:15:19.132201 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.131713 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" event={"ID":"fc146841-dec1-42b0-b593-61c9d0e61b6c","Type":"ContainerDied","Data":"84333229cb398638bc6d8de19ddd01984972309c0ae03e7b74eddc055fc283eb"} Oct 30 19:15:19.133367 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.133346 4582 generic.go:356] "Generic (PLEG): container finished" podID="c02c7ed0-c765-49e2-8cc8-0d7636c688e5" containerID="b9ec3fc720d32f0482c7296d1a3065a487a593b4c7503f07a0c56812d7e2245c" exitCode=0 Oct 30 19:15:19.133450 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.133426 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" event={"ID":"c02c7ed0-c765-49e2-8cc8-0d7636c688e5","Type":"ContainerDied","Data":"b9ec3fc720d32f0482c7296d1a3065a487a593b4c7503f07a0c56812d7e2245c"} Oct 30 19:15:19.134673 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.134654 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" Oct 30 19:15:19.134781 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.134677 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w" event={"ID":"c05304f3-6bf0-4034-9961-58878f20c73d","Type":"ContainerDied","Data":"58561627c14a3f6788e2674a80d20b660664838eef4580a0cd91a722ac2c9a06"} Oct 30 19:15:19.134781 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.134726 4582 scope.go:122] "RemoveContainer" containerID="26ec0acd306939ad1b9a279f753fef3bf6b2c95f8b8173dd3f663b589b152c6e" Oct 30 19:15:19.136904 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.136886 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" Oct 30 19:15:19.137009 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.136901 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7505801d-q82xr" event={"ID":"bcff78ab-c780-4e1b-8cde-89e20217a12c","Type":"ContainerDied","Data":"49545f18c781a5948708356c4a8380cab58cb393b6ccbe293957b26e34233926"} Oct 30 19:15:19.137009 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.136939 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49545f18c781a5948708356c4a8380cab58cb393b6ccbe293957b26e34233926" Oct 30 19:15:19.141686 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.141660 4582 generic.go:356] "Generic (PLEG): container finished" podID="5cbadbcc-562f-40be-846a-c21840a16eac" containerID="3a766ff470654838d377e199ea2b5ef083044975de928fad5867f100cd995d5d" exitCode=60 Oct 30 19:15:19.141794 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.141728 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" event={"ID":"5cbadbcc-562f-40be-846a-c21840a16eac","Type":"ContainerDied","Data":"3a766ff470654838d377e199ea2b5ef083044975de928fad5867f100cd995d5d"} Oct 30 19:15:19.142087 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.142067 4582 scope.go:122] "RemoveContainer" containerID="3a766ff470654838d377e199ea2b5ef083044975de928fad5867f100cd995d5d" Oct 30 19:15:19.142303 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:19.142283 4582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-qhklx_odh-ai-gateway-infra(5cbadbcc-562f-40be-846a-c21840a16eac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" Oct 30 19:15:19.145478 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.145460 4582 scope.go:122] "RemoveContainer" containerID="3c16f73f207c135972c2cb5fb6beea76e35ac1e4f019902d82da155736881e15" Oct 30 19:15:19.204684 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.204658 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w"] Oct 30 19:15:19.206964 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.206941 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889795-kg85w"] Oct 30 19:15:19.243282 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.243258 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx"] Oct 30 19:15:19.300036 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.300010 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d1d4fc-e4d1-4a19-9b20-beea70199eea" path="/var/lib/kubelet/pods/16d1d4fc-e4d1-4a19-9b20-beea70199eea/volumes" Oct 30 19:15:19.300404 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:19.300392 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" path="/var/lib/kubelet/pods/c05304f3-6bf0-4034-9961-58878f20c73d/volumes" Oct 30 19:15:20.341654 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.341628 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:20.344264 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.344243 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:20.347468 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.347451 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:20.492807 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.492728 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-kube-api-access-5928k\" (UniqueName: \"kubernetes.io/projected/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-kube-api-access-5928k\") pod \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " Oct 30 19:15:20.492807 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.492768 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-xks-ca\") pod \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " Oct 30 19:15:20.492807 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.492805 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc146841-dec1-42b0-b593-61c9d0e61b6c-kube-api-access-lnb6w\" (UniqueName: \"kubernetes.io/projected/fc146841-dec1-42b0-b593-61c9d0e61b6c-kube-api-access-lnb6w\") pod \"fc146841-dec1-42b0-b593-61c9d0e61b6c\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " Oct 30 19:15:20.493126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.492844 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-xks-ca\") pod \"fc146841-dec1-42b0-b593-61c9d0e61b6c\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " Oct 30 19:15:20.493126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.492873 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-openshift-ca\") pod \"fc146841-dec1-42b0-b593-61c9d0e61b6c\" (UID: \"fc146841-dec1-42b0-b593-61c9d0e61b6c\") " Oct 30 19:15:20.493126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.492975 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbadbcc-562f-40be-846a-c21840a16eac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbadbcc-562f-40be-846a-c21840a16eac-maas-api-service-ca\") pod \"5cbadbcc-562f-40be-846a-c21840a16eac\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " Oct 30 19:15:20.493126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.493016 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbadbcc-562f-40be-846a-c21840a16eac-kube-api-access-g9jwl\" (UniqueName: \"kubernetes.io/projected/5cbadbcc-562f-40be-846a-c21840a16eac-kube-api-access-g9jwl\") pod \"5cbadbcc-562f-40be-846a-c21840a16eac\" (UID: \"5cbadbcc-562f-40be-846a-c21840a16eac\") " Oct 30 19:15:20.493126 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.493049 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-openshift-ca\") pod \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\" (UID: \"c02c7ed0-c765-49e2-8cc8-0d7636c688e5\") " Oct 30 19:15:20.493770 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.493704 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-openshift-ca" pod "fc146841-dec1-42b0-b593-61c9d0e61b6c" (UID: "fc146841-dec1-42b0-b593-61c9d0e61b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.493770 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.493727 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-openshift-ca" pod "c02c7ed0-c765-49e2-8cc8-0d7636c688e5" (UID: "c02c7ed0-c765-49e2-8cc8-0d7636c688e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.494034 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.493888 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbadbcc-562f-40be-846a-c21840a16eac-maas-api-service-ca" pod "5cbadbcc-562f-40be-846a-c21840a16eac" (UID: "5cbadbcc-562f-40be-846a-c21840a16eac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.495845 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.495791 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc146841-dec1-42b0-b593-61c9d0e61b6c-kube-api-access-lnb6w" pod "fc146841-dec1-42b0-b593-61c9d0e61b6c" (UID: "fc146841-dec1-42b0-b593-61c9d0e61b6c"). InnerVolumeSpecName "kube-api-access-lnb6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.496152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.496119 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbadbcc-562f-40be-846a-c21840a16eac-kube-api-access-g9jwl" pod "5cbadbcc-562f-40be-846a-c21840a16eac" (UID: "5cbadbcc-562f-40be-846a-c21840a16eac"). InnerVolumeSpecName "kube-api-access-g9jwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.496850 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.496756 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-xks-ca" pod "fc146841-dec1-42b0-b593-61c9d0e61b6c" (UID: "fc146841-dec1-42b0-b593-61c9d0e61b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:20.496957 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.496866 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-kube-api-access-5928k" pod "c02c7ed0-c765-49e2-8cc8-0d7636c688e5" (UID: "c02c7ed0-c765-49e2-8cc8-0d7636c688e5"). InnerVolumeSpecName "kube-api-access-5928k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.498084 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.498065 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-xks-ca" pod "c02c7ed0-c765-49e2-8cc8-0d7636c688e5" (UID: "c02c7ed0-c765-49e2-8cc8-0d7636c688e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:20.595135 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595102 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbadbcc-562f-40be-846a-c21840a16eac-maas-api-service-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.595135 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595131 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jwl\" (UniqueName: \"kubernetes.io/projected/5cbadbcc-562f-40be-846a-c21840a16eac-kube-api-access-g9jwl\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.595135 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595142 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.595360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595150 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5928k\" (UniqueName: \"kubernetes.io/projected/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-kube-api-access-5928k\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.595360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595160 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02c7ed0-c765-49e2-8cc8-0d7636c688e5-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.595360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595169 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnb6w\" (UniqueName: \"kubernetes.io/projected/fc146841-dec1-42b0-b593-61c9d0e61b6c-kube-api-access-lnb6w\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.595360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595179 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-xks-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.595360 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.595187 4582 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc146841-dec1-42b0-b593-61c9d0e61b6c-maas-api-openshift-ca\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:20.597242 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:20.597214 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:21.157670 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.157639 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" Oct 30 19:15:21.157876 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.157679 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx" event={"ID":"5cbadbcc-562f-40be-846a-c21840a16eac","Type":"ContainerDied","Data":"14cd12cbb052db780576c895c3ac7aa14f5d98989e6dfda5c5929f1090c421d9"} Oct 30 19:15:21.157876 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.157754 4582 scope.go:122] "RemoveContainer" containerID="3a766ff470654838d377e199ea2b5ef083044975de928fad5867f100cd995d5d" Oct 30 19:15:21.159112 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.158984 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" Oct 30 19:15:21.159112 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.159003 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683ae185-nkdql" event={"ID":"fc146841-dec1-42b0-b593-61c9d0e61b6c","Type":"ContainerDied","Data":"170fd0990eaee4e772c0461437c5afb315248b48947a46194369fcd0306256df"} Oct 30 19:15:21.159112 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.159062 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170fd0990eaee4e772c0461437c5afb315248b48947a46194369fcd0306256df" Oct 30 19:15:21.160281 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.160263 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" Oct 30 19:15:21.160401 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.160261 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac36549-fnmr9" event={"ID":"c02c7ed0-c765-49e2-8cc8-0d7636c688e5","Type":"ContainerDied","Data":"655969604aa4134b91f0b5905f6c132cd57d48be8a3f50e7c840381aca30b8e8"} Oct 30 19:15:21.160401 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.160398 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="655969604aa4134b91f0b5905f6c132cd57d48be8a3f50e7c840381aca30b8e8" Oct 30 19:15:21.238152 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.238123 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx"] Oct 30 19:15:21.246274 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.246241 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-qhklx"] Oct 30 19:15:21.299746 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:21.299715 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" path="/var/lib/kubelet/pods/5cbadbcc-562f-40be-846a-c21840a16eac/volumes" Oct 30 19:15:24.931891 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:24.931851 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" containerID="cri-o://fcee0131667c1b5b8db115600162216fbae06301bd754136319b522433e2ffc1" gracePeriod=45 Oct 30 19:15:25.185511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.185447 4582 generic.go:356] "Generic (PLEG): container finished" podID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerID="fcee0131667c1b5b8db115600162216fbae06301bd754136319b522433e2ffc1" exitCode=0 Oct 30 19:15:25.185511 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.185501 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" event={"ID":"8cad11d6-f33d-4ab8-8548-3e07f62429b0","Type":"ContainerDied","Data":"fcee0131667c1b5b8db115600162216fbae06301bd754136319b522433e2ffc1"} Oct 30 19:15:25.185665 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.185530 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" event={"ID":"8cad11d6-f33d-4ab8-8548-3e07f62429b0","Type":"ContainerDied","Data":"4aa6f38b3d2353148c6e7ebc453e5d9075f5d8d8072be265db4636201142faa3"} Oct 30 19:15:25.185665 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.185541 4582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa6f38b3d2353148c6e7ebc453e5d9075f5d8d8072be265db4636201142faa3" Oct 30 19:15:25.187155 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.187137 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 19:15:25.241246 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241215 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tmp-dir\") pod \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " Oct 30 19:15:25.241439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241279 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tls-certs\") pod \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " Oct 30 19:15:25.241439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241347 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-model-cache\") pod \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " Oct 30 19:15:25.241439 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241403 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-dshm\") pod \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " Oct 30 19:15:25.241616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241439 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-home\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-home\") pod \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " Oct 30 19:15:25.241616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241494 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cad11d6-f33d-4ab8-8548-3e07f62429b0-kube-api-access-smzc2\" (UniqueName: \"kubernetes.io/projected/8cad11d6-f33d-4ab8-8548-3e07f62429b0-kube-api-access-smzc2\") pod \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\" (UID: \"8cad11d6-f33d-4ab8-8548-3e07f62429b0\") " Oct 30 19:15:25.241616 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241538 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tmp-dir" pod "8cad11d6-f33d-4ab8-8548-3e07f62429b0" (UID: "8cad11d6-f33d-4ab8-8548-3e07f62429b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.241786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241660 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-model-cache" pod "8cad11d6-f33d-4ab8-8548-3e07f62429b0" (UID: "8cad11d6-f33d-4ab8-8548-3e07f62429b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.241786 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.241760 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-home" pod "8cad11d6-f33d-4ab8-8548-3e07f62429b0" (UID: "8cad11d6-f33d-4ab8-8548-3e07f62429b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.242477 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.242453 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.242588 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.242479 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.242588 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.242518 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.243788 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.243765 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-dshm" pod "8cad11d6-f33d-4ab8-8548-3e07f62429b0" (UID: "8cad11d6-f33d-4ab8-8548-3e07f62429b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.244038 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.244012 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tls-certs" pod "8cad11d6-f33d-4ab8-8548-3e07f62429b0" (UID: "8cad11d6-f33d-4ab8-8548-3e07f62429b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:25.244136 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.244078 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cad11d6-f33d-4ab8-8548-3e07f62429b0-kube-api-access-smzc2" pod "8cad11d6-f33d-4ab8-8548-3e07f62429b0" (UID: "8cad11d6-f33d-4ab8-8548-3e07f62429b0"). InnerVolumeSpecName "kube-api-access-smzc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:25.343249 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.343215 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cad11d6-f33d-4ab8-8548-3e07f62429b0-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.343249 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.343247 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cad11d6-f33d-4ab8-8548-3e07f62429b0-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.343424 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.343258 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzc2\" (UniqueName: \"kubernetes.io/projected/8cad11d6-f33d-4ab8-8548-3e07f62429b0-kube-api-access-smzc2\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.983735 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:25.983673 4582 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" containerID="cri-o://701b5d1a8d8ae8e279d30b5921be7a9f764bfb79ecc93281eb5edf8be6398ee9" gracePeriod=45 Oct 30 19:15:26.193114 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.193074 4582 generic.go:356] "Generic (PLEG): container finished" podID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerID="701b5d1a8d8ae8e279d30b5921be7a9f764bfb79ecc93281eb5edf8be6398ee9" exitCode=0 Oct 30 19:15:26.193277 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.193134 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" event={"ID":"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444","Type":"ContainerDied","Data":"701b5d1a8d8ae8e279d30b5921be7a9f764bfb79ecc93281eb5edf8be6398ee9"} Oct 30 19:15:26.193277 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.193197 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4" Oct 30 19:15:26.237231 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.237174 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 19:15:26.243729 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.243706 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4"] Oct 30 19:15:26.251658 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.251636 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tls-certs\" (UniqueName: \"kubernetes.io/secret/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tls-certs\") pod \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " Oct 30 19:15:26.251753 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.251691 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-dshm\") pod \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " Oct 30 19:15:26.251753 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.251732 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-home\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-home\") pod \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " Oct 30 19:15:26.251906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.251795 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-kube-api-access-sqd45\" (UniqueName: \"kubernetes.io/projected/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-kube-api-access-sqd45\") pod \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " Oct 30 19:15:26.251906 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.251890 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tmp-dir\") pod \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " Oct 30 19:15:26.252017 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.251965 4582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-model-cache\") pod \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\" (UID: \"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444\") " Oct 30 19:15:26.252075 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.252032 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-home" pod "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" (UID: "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.252268 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.252241 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tmp-dir" pod "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" (UID: "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.252268 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.252251 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-model-cache" pod "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" (UID: "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.252446 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.252346 4582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-home\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.252446 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.252366 4582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tmp-dir\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.252446 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.252382 4582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-model-cache\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.254107 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.254079 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-kube-api-access-sqd45" pod "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" (UID: "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444"). InnerVolumeSpecName "kube-api-access-sqd45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:26.254205 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.254184 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-dshm" pod "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" (UID: "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.254255 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.254193 4582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tls-certs" pod "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" (UID: "edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:26.255598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.255579 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/premium-w2-45dc34-kserve-57d48d4b87-wt4q4"] Oct 30 19:15:26.353598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.353565 4582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqd45\" (UniqueName: \"kubernetes.io/projected/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-kube-api-access-sqd45\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.353598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.353593 4582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-tls-certs\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.353598 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:26.353602 4582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444-dshm\") on node \"ip-10-0-133-124.ec2.internal\" DevicePath \"\"" Oct 30 19:15:27.199887 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:27.199857 4582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" Oct 30 19:15:27.200358 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:27.199879 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7" event={"ID":"edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444","Type":"ContainerDied","Data":"d9071fe27565c545f1bca2b1b146e108dc61f182129b02f4108f86f926a74af3"} Oct 30 19:15:27.200358 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:27.199935 4582 scope.go:122] "RemoveContainer" containerID="701b5d1a8d8ae8e279d30b5921be7a9f764bfb79ecc93281eb5edf8be6398ee9" Oct 30 19:15:27.253024 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:27.252991 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7"] Oct 30 19:15:27.265182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:27.265152 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-921450/premium-w0-921450-kserve-84bdb6f7c5-7kbr7"] Oct 30 19:15:27.299156 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:27.299120 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" path="/var/lib/kubelet/pods/8cad11d6-f33d-4ab8-8548-3e07f62429b0/volumes" Oct 30 19:15:27.299599 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:27.299579 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" path="/var/lib/kubelet/pods/edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444/volumes" Oct 30 19:15:29.921999 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:29.921962 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:30.494995 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:30.494964 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:30.627047 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:30.627006 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:31.024902 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.024863 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:31.126098 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.126067 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:31.186580 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.186545 4582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj"] Oct 30 19:15:31.198346 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.198318 4582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-45dc34-s8grj"] Oct 30 19:15:31.299646 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.299573 4582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1fe9f0-55d7-4818-ab9a-9516f2820145" path="/var/lib/kubelet/pods/0b1fe9f0-55d7-4818-ab9a-9516f2820145/volumes" Oct 30 19:15:31.655888 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.655856 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:31.804584 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.804556 4582 scope.go:122] "RemoveContainer" containerID="fcee0131667c1b5b8db115600162216fbae06301bd754136319b522433e2ffc1" Oct 30 19:15:31.813367 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:31.813345 4582 scope.go:122] "RemoveContainer" containerID="b691225f9ba9e06dfa530b6507f6d20de4362a9f8164fea3365ffcc442a6988c" Oct 30 19:15:33.772721 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:33.772690 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:34.158485 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:34.158447 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:34.293024 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:34.292996 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:34.391067 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:34.391034 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:34.687044 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:34.686998 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:34.793052 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:34.793018 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:34.891627 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:34.891592 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:35.341254 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:35.341225 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:36.227467 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:15:36.227434 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:15:36.616917 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:36.616799 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:36.717290 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:36.717249 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:37.945744 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:37.945710 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:39.673729 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:39.673692 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:39.812131 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:39.812097 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:40.060436 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:40.060350 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.091018 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:43.090984 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.215006 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:43.214976 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.443508 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:43.443471 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.664068 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:43.664037 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.791467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:43.791382 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:44.046446 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:44.046363 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:44.961323 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:44.961286 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:45.680841 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:45.680790 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dhkgf_3e0ef342-18fe-4c80-9911-00af0f03669b/kuadrant-console-plugin/0.log" Oct 30 19:15:47.632968 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:47.632931 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:47.757041 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:47.757007 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:47.899492 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:47.899403 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:48.048101 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:48.048068 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:48.352252 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:48.352178 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:15:48.603654 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:48.603585 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:15:49.775577 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:49.775544 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:50.110719 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:50.110643 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:55.829190 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:55.829152 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:55.928842 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:55.928792 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:56.032800 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:15:56.032752 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:06.223942 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:16:06.223905 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:16:07.733303 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:07.733238 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:08.453128 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:08.453083 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:10.749994 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:10.749957 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:16:20.800704 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:20.800666 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:36.224217 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:16:36.224178 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:16:43.712171 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:43.712138 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:16:43.923564 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:43.923529 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:16:45.275039 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:45.275004 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:16:48.218554 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:48.218524 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-683ae185-nkdql_fc146841-dec1-42b0-b593-61c9d0e61b6c/revoke-keys/0.log" Oct 30 19:16:48.416242 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:48.416217 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7505801d-q82xr_bcff78ab-c780-4e1b-8cde-89e20217a12c/revoke-keys/0.log" Oct 30 19:16:49.780337 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:49.780297 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dac36549-fnmr9_c02c7ed0-c765-49e2-8cc8-0d7636c688e5/revoke-keys/0.log" Oct 30 19:16:52.322112 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:52.322081 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dhkgf_3e0ef342-18fe-4c80-9911-00af0f03669b/kuadrant-console-plugin/0.log" Oct 30 19:16:55.416485 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:55.416447 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-pkgv6_d6299316-1bf3-4b66-99fa-55fcd37b64f2/kube-auth-proxy/0.log" Oct 30 19:16:55.998690 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:55.998656 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-5cfxk_cd66eba8-79c3-4cb5-8b81-ac65cac3619d/istio-proxy/0.log" Oct 30 19:16:56.201391 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:56.201360 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-tpnm7_70547fc1-ceea-47fd-abb0-c740d151c84c/istio-proxy/0.log" Oct 30 19:16:56.568396 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:56.568367 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6c557_f0d6333d-538f-47bd-9240-f13efd118817/payload-pre-processing/0.log" Oct 30 19:16:56.752945 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:56.752910 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log" Oct 30 19:16:57.743885 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:57.743854 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bx4v6_8e5667fa-9807-46ac-80e6-ab902edfb512/main/0.log" Oct 30 19:16:58.056345 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:58.056268 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.298859 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:58.298540 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-nscmh_d0f4a581-7366-46ab-9f04-eebb2f9ca897/main/0.log" Oct 30 19:16:58.504646 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:58.504612 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8clmp5_ed7bf2c0-385c-4a64-99fb-0913b3463bfa/main/0.log" Oct 30 19:16:58.840508 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:58.840431 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.888234 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:16:58.888211 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kq2gj_bad83681-b01e-416a-bd8a-fafae54c4f56/main/0.log" Oct 30 19:17:06.223499 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:17:06.223463 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:17:15.642725 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:15.642686 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:15.822897 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:15.822860 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:19.330486 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:19.330451 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:19.543974 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:19.543933 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:20.833477 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:20.833447 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:25.190315 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:25.190283 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jg28z_845bfa8a-e757-4cd4-bcd0-f3625bb21afa/global-pull-secret-syncer/0.log" Oct 30 19:17:26.070148 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:26.070094 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" Oct 30 19:17:26.700160 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:26.700124 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-124.ec2.internal_2c67125f85ae263b41b821237802c161/haproxy/0.log" Oct 30 19:17:31.637523 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:31.637492 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:17:31.725645 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:31.725613 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:17:33.163122 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:33.163094 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:17:35.027327 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:35.027297 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dhkgf_3e0ef342-18fe-4c80-9911-00af0f03669b/kuadrant-console-plugin/0.log" Oct 30 19:17:35.468294 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:35.468246 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:17:35.652335 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:35.652303 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:17:36.227897 ip-10-0-133-124 kubenswrapper[4582]: E1030 19:17:36.227860 4582 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d6r5" containerName="extractor" Oct 30 19:17:36.686665 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.686613 4582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q"] Oct 30 19:17:36.687212 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687192 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687214 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687228 4582 cpu_manager.go:414] "containerMap: removing container" podUID="fc146841-dec1-42b0-b593-61c9d0e61b6c" containerName="revoke-keys" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687234 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc146841-dec1-42b0-b593-61c9d0e61b6c" containerName="revoke-keys" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687246 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687251 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687259 4582 cpu_manager.go:414] "containerMap: removing container" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687264 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687273 4582 cpu_manager.go:414] "containerMap: removing container" podUID="bcff78ab-c780-4e1b-8cde-89e20217a12c" containerName="revoke-keys" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687277 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcff78ab-c780-4e1b-8cde-89e20217a12c" containerName="revoke-keys" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687290 4582 cpu_manager.go:414] "containerMap: removing container" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687296 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687303 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687308 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687318 4582 cpu_manager.go:414] "containerMap: removing container" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" Oct 30 19:17:36.687320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687323 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687329 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687334 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687344 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687349 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687355 4582 cpu_manager.go:414] "containerMap: removing container" podUID="c02c7ed0-c765-49e2-8cc8-0d7636c688e5" containerName="revoke-keys" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687360 4582 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02c7ed0-c765-49e2-8cc8-0d7636c688e5" containerName="revoke-keys" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687439 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc146841-dec1-42b0-b593-61c9d0e61b6c" containerName="revoke-keys" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687451 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687456 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbadbcc-562f-40be-846a-c21840a16eac" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687466 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcff78ab-c780-4e1b-8cde-89e20217a12c" containerName="revoke-keys" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687474 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="edaf5b8e-7d10-4b33-8e77-5dcc2ccf6444" containerName="main" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687483 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687499 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cad11d6-f33d-4ab8-8548-3e07f62429b0" containerName="main" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687509 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05304f3-6bf0-4034-9961-58878f20c73d" containerName="cleanup" Oct 30 19:17:36.688182 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.687519 4582 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02c7ed0-c765-49e2-8cc8-0d7636c688e5" containerName="revoke-keys" Oct 30 19:17:36.691149 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.691130 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.708597 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.708566 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q"] Oct 30 19:17:36.718495 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.718467 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj67d\"/\"openshift-service-ca.crt\"" Oct 30 19:17:36.719963 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.719941 4582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wj67d\"/\"default-dockercfg-g4gj2\"" Oct 30 19:17:36.720123 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.719965 4582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj67d\"/\"kube-root-ca.crt\"" Oct 30 19:17:36.804299 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.804268 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-lib-modules\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.804582 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.804314 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-proc\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.804582 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.804397 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vxt\" (UniqueName: \"kubernetes.io/projected/5f012fc9-375e-42cd-abc0-3d65aefb74dc-kube-api-access-c2vxt\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.804582 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.804432 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-podres\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.804582 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.804452 4582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-sys\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.904904 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.904872 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-podres\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.904904 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.904911 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-sys\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.905125 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.904984 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-lib-modules\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.905125 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.905007 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-podres\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.905125 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.905057 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-sys\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.905125 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.905089 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-proc\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.905261 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.905135 4582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vxt\" (UniqueName: \"kubernetes.io/projected/5f012fc9-375e-42cd-abc0-3d65aefb74dc-kube-api-access-c2vxt\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.905261 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.905091 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-lib-modules\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.905261 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.905148 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f012fc9-375e-42cd-abc0-3d65aefb74dc-proc\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.906956 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.906937 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:17:36.924643 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.924618 4582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vxt\" (UniqueName: \"kubernetes.io/projected/5f012fc9-375e-42cd-abc0-3d65aefb74dc-kube-api-access-c2vxt\") pod \"perf-node-gather-daemonset-6dr6q\" (UID: \"5f012fc9-375e-42cd-abc0-3d65aefb74dc\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:36.995044 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:36.994976 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:17:37.005320 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.005301 4582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:37.209197 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.209155 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:17:37.302909 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.302885 4582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q"] Oct 30 19:17:37.306138 ip-10-0-133-124 kubenswrapper[4582]: W1030 19:17:37.306099 4582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f012fc9_375e_42cd_abc0_3d65aefb74dc.slice/crio-476b7b126f381f999e9500d4bce5753a0c757542e5e0c6fbc2f98ba3a03e7326 WatchSource:0}: Error finding container 476b7b126f381f999e9500d4bce5753a0c757542e5e0c6fbc2f98ba3a03e7326: Status 404 returned error can't find the container with id 476b7b126f381f999e9500d4bce5753a0c757542e5e0c6fbc2f98ba3a03e7326 Oct 30 19:17:37.856527 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.856491 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" event={"ID":"5f012fc9-375e-42cd-abc0-3d65aefb74dc","Type":"ContainerStarted","Data":"32b600c04e344f7b7c5b1d1fd6b900b68769b1bc6767b34b5efa91e2f21c96f9"} Oct 30 19:17:37.856722 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.856592 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:37.856722 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.856626 4582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" event={"ID":"5f012fc9-375e-42cd-abc0-3d65aefb74dc","Type":"ContainerStarted","Data":"476b7b126f381f999e9500d4bce5753a0c757542e5e0c6fbc2f98ba3a03e7326"} Oct 30 19:17:37.876936 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.876909 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956cfd2c-560c-4b7c-8484-1334cf3b10bf/alertmanager/0.log" Oct 30 19:17:37.950963 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:37.950904 4582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" podStartSLOduration=1.9508915980000001 podStartE2EDuration="1.950891598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:37.948450145 +0000 UTC m=+4754.772844438" watchObservedRunningTime="2026-10-30 19:17:37.950891598 +0000 UTC m=+4754.775285938" Oct 30 19:17:38.068945 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:38.068903 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956cfd2c-560c-4b7c-8484-1334cf3b10bf/config-reloader/0.log" Oct 30 19:17:38.197403 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:38.197374 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956cfd2c-560c-4b7c-8484-1334cf3b10bf/kube-rbac-proxy-web/0.log" Oct 30 19:17:38.324473 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:38.324445 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956cfd2c-560c-4b7c-8484-1334cf3b10bf/kube-rbac-proxy/0.log" Oct 30 19:17:38.420181 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:38.420157 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:17:38.453727 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:38.453668 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956cfd2c-560c-4b7c-8484-1334cf3b10bf/kube-rbac-proxy-metric/0.log" Oct 30 19:17:38.578596 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:38.578566 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956cfd2c-560c-4b7c-8484-1334cf3b10bf/prom-label-proxy/0.log" Oct 30 19:17:38.710532 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:38.710460 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_956cfd2c-560c-4b7c-8484-1334cf3b10bf/init-config-reloader/0.log" Oct 30 19:17:39.134162 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:39.134134 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:17:39.185241 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:39.185214 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:17:39.603481 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:39.603403 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-trwdq_5e1f58b2-45e1-420b-be68-f6ff9e2eb0c4/monitoring-plugin/0.log" Oct 30 19:17:40.019105 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:40.019077 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:17:40.081969 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:40.081942 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:17:40.461557 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:40.461525 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:17:40.689031 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:40.689000 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zxgbz_f089c1a2-ca6d-486d-91b5-1d60a0d5d262/node-exporter/0.log" Oct 30 19:17:40.815841 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:40.815741 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zxgbz_f089c1a2-ca6d-486d-91b5-1d60a0d5d262/kube-rbac-proxy/0.log" Oct 30 19:17:40.951145 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:40.951118 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zxgbz_f089c1a2-ca6d-486d-91b5-1d60a0d5d262/init-textfile/0.log" Oct 30 19:17:41.083103 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.083033 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-999c6_b3b57703-781e-4309-a8de-db9cc9a63157/kube-rbac-proxy-main/0.log" Oct 30 19:17:41.206669 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.206641 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-999c6_b3b57703-781e-4309-a8de-db9cc9a63157/kube-rbac-proxy-self/0.log" Oct 30 19:17:41.335490 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.335422 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-999c6_b3b57703-781e-4309-a8de-db9cc9a63157/openshift-state-metrics/0.log" Oct 30 19:17:41.475836 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.475804 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b967cbbc-55c5-4128-84de-19b842802961/prometheus/0.log" Oct 30 19:17:41.496334 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.496309 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:17:41.616356 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.616295 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b967cbbc-55c5-4128-84de-19b842802961/config-reloader/0.log" Oct 30 19:17:41.766096 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.766070 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b967cbbc-55c5-4128-84de-19b842802961/thanos-sidecar/0.log" Oct 30 19:17:41.895157 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:41.895134 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b967cbbc-55c5-4128-84de-19b842802961/kube-rbac-proxy-web/0.log" Oct 30 19:17:42.025976 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:42.025949 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b967cbbc-55c5-4128-84de-19b842802961/kube-rbac-proxy/0.log" Oct 30 19:17:42.167205 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:42.167135 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b967cbbc-55c5-4128-84de-19b842802961/kube-rbac-proxy-thanos/0.log" Oct 30 19:17:42.313259 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:42.313227 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b967cbbc-55c5-4128-84de-19b842802961/init-config-reloader/0.log" Oct 30 19:17:42.712188 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:42.712161 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-vpfj4_f0119471-101c-491b-8a67-ca0c104d2261/prometheus-operator-admission-webhook/0.log" Oct 30 19:17:42.853467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:42.853435 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65db959845-drtds_ef2f4767-ef82-4e82-bba6-3857a52f3a3d/telemeter-client/0.log" Oct 30 19:17:42.981202 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:42.981122 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65db959845-drtds_ef2f4767-ef82-4e82-bba6-3857a52f3a3d/reload/0.log" Oct 30 19:17:43.112772 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.112745 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65db959845-drtds_ef2f4767-ef82-4e82-bba6-3857a52f3a3d/kube-rbac-proxy/0.log" Oct 30 19:17:43.166251 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.166217 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b665f6876-csbcr_a7d9c49e-a7f9-44c4-9c8a-cd0d77da88e4/manager/0.log" Oct 30 19:17:43.205634 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.205606 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-sbh4k_e141d889-3d52-4eb2-8eb5-6214d5078068/manager/0.log" Oct 30 19:17:43.249768 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.249704 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b56887585-wwl24_a2f81431-fa44-44fa-8cee-2d7afb4323be/thanos-query/0.log" Oct 30 19:17:43.374225 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.374191 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b56887585-wwl24_a2f81431-fa44-44fa-8cee-2d7afb4323be/kube-rbac-proxy-web/0.log" Oct 30 19:17:43.505699 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.505618 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b56887585-wwl24_a2f81431-fa44-44fa-8cee-2d7afb4323be/kube-rbac-proxy/0.log" Oct 30 19:17:43.643404 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.643334 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b56887585-wwl24_a2f81431-fa44-44fa-8cee-2d7afb4323be/prom-label-proxy/0.log" Oct 30 19:17:43.776306 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.776211 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b56887585-wwl24_a2f81431-fa44-44fa-8cee-2d7afb4323be/kube-rbac-proxy-rules/0.log" Oct 30 19:17:43.874305 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.874275 4582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-6dr6q" Oct 30 19:17:43.913467 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:43.913442 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b56887585-wwl24_a2f81431-fa44-44fa-8cee-2d7afb4323be/kube-rbac-proxy-metrics/0.log" Oct 30 19:17:44.750449 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:44.750412 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-59st9_58c0f995-78f8-4a88-9cef-7c976677ebdc/server/0.log" Oct 30 19:17:45.633453 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:45.633422 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-pkgv6_d6299316-1bf3-4b66-99fa-55fcd37b64f2/kube-auth-proxy/0.log" Oct 30 19:17:45.866997 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:45.866968 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-5cfxk_cd66eba8-79c3-4cb5-8b81-ac65cac3619d/istio-proxy/0.log" Oct 30 19:17:45.963829 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:45.963736 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-tpnm7_70547fc1-ceea-47fd-abb0-c740d151c84c/istio-proxy/0.log" Oct 30 19:17:46.105522 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:46.105490 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6c557_f0d6333d-538f-47bd-9240-f13efd118817/payload-pre-processing/0.log" Oct 30 19:17:46.175410 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:46.175380 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log" Oct 30 19:17:48.878873 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:48.878843 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-2g2gm_24c78f02-e9ed-4344-b8fc-9a20c413548d/download-server/0.log" Oct 30 19:17:49.531532 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:49.531497 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-pkgv6_d6299316-1bf3-4b66-99fa-55fcd37b64f2/kube-auth-proxy/0.log" Oct 30 19:17:49.740497 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:49.740466 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-5cfxk_cd66eba8-79c3-4cb5-8b81-ac65cac3619d/istio-proxy/0.log" Oct 30 19:17:49.810398 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:49.810324 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-tpnm7_70547fc1-ceea-47fd-abb0-c740d151c84c/istio-proxy/0.log" Oct 30 19:17:49.945700 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:49.945674 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6c557_f0d6333d-538f-47bd-9240-f13efd118817/payload-pre-processing/0.log" Oct 30 19:17:49.949920 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:49.949895 4582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtxd_e6c5c437-9fa5-4d3a-8ba5-ed7b6ff6656a/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:50.012946 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:50.012922 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log" Oct 30 19:17:52.262928 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:52.262900 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dhkgf_3e0ef342-18fe-4c80-9911-00af0f03669b/kuadrant-console-plugin/0.log" Oct 30 19:17:53.085641 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:53.085608 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmstz_3f0329ae-a289-4958-83b9-ff0c5455c068/dns/0.log" Oct 30 19:17:53.259858 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:53.259814 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmstz_3f0329ae-a289-4958-83b9-ff0c5455c068/kube-rbac-proxy/0.log" Oct 30 19:17:53.663437 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:53.663411 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jrtkt_6f7d3975-f96e-4f38-9535-1f37d2142493/dns-node-resolver/0.log" Oct 30 19:17:55.219887 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:55.219858 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dhkgf_3e0ef342-18fe-4c80-9911-00af0f03669b/kuadrant-console-plugin/0.log" Oct 30 19:17:55.823768 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:55.823739 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rfjbn_f89b785d-9cd4-48ed-a4f7-85b5ddcd086c/node-ca/0.log" Oct 30 19:17:55.934006 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:55.933975 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-pkgv6_d6299316-1bf3-4b66-99fa-55fcd37b64f2/kube-auth-proxy/0.log" Oct 30 19:17:56.140886 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:56.140861 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-5cfxk_cd66eba8-79c3-4cb5-8b81-ac65cac3619d/istio-proxy/0.log" Oct 30 19:17:56.202630 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:56.202605 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-tpnm7_70547fc1-ceea-47fd-abb0-c740d151c84c/istio-proxy/0.log" Oct 30 19:17:56.326980 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:56.326947 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6c557_f0d6333d-538f-47bd-9240-f13efd118817/payload-pre-processing/0.log" Oct 30 19:17:56.387100 ip-10-0-133-124 kubenswrapper[4582]: I1030 19:17:56.387071 4582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-97rs5_70164f26-3f3a-42b1-be62-923b183ee563/payload-processing/0.log"